{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 第二题：神经网络：线性回归"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "实验内容：\n",
    "1. 学会梯度下降的基本思想\n",
    "2. 学会使用梯度下降求解线性回归\n",
    "3. 了解归一化处理的作用"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 线性回归\n",
    "\n",
    "<img src=\"https://davidham3.github.io/blog/2018/09/11/logistic-regression/Fig0.png\" width=300>\n",
    "\n",
    "我们来完成最简单的线性回归，上图是一个最简单的神经网络，一个输入层，一个输出层，没有激活函数。  \n",
    "我们记输入为$X \\in \\mathbb{R}^{n \\times m}$，输出为$Z \\in \\mathbb{R}^{n}$。输入包含了$n$个样本，$m$个特征，输出是对这$n$个样本的预测值。  \n",
    "输入层到输出层的权重和偏置，我们记为$W \\in \\mathbb{R}^{m}$和$b \\in \\mathbb{R}$。  \n",
    "输出层没有激活函数，所以上面的神经网络的前向传播过程写为：\n",
    "\n",
    "$$\n",
    "Z = XW + b\n",
    "$$\n",
    "\n",
    "我们使用均方误差作为模型的损失函数\n",
    "\n",
    "$$\n",
    "\\mathrm{loss}(y, \\hat{y}) = \\frac{1}{n} \\sum^n_{i=1}(y_i - \\hat{y_i})^2\n",
    "$$\n",
    "\n",
    "我们通过调整参数$W$和$b$来降低均方误差，或者说是以降低均方误差为目标，学习参数$W$和参数$b$。当均方误差下降的时候，我们认为当前的模型的预测值$Z$与真值$y$越来越接近，也就是说模型正在学习如何让自己的预测值变得更准确。\n",
    "\n",
    "在前面的课程中，我们已经学习了这种线性回归模型可以使用最小二乘法求解，最小二乘法在求解数据量较小的问题的时候很有效，但是最小二乘法的时间复杂度很高，一旦数据量变大，效率很低，实际应用中我们会使用梯度下降等基于梯度的优化算法来求解参数$W$和参数$b$。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 梯度下降\n",
    "\n",
    "梯度下降是一种常用的优化算法，通俗来说就是计算出参数的梯度（损失函数对参数的偏导数的导数值），然后将参数减去参数的梯度乘以一个很小的数（下面的公式），来改变参数，然后重新计算损失函数，再次计算梯度，再次进行调整，通过一定次数的迭代，参数就会收敛到最优点附近。\n",
    "\n",
    "在我们的这个线性回归问题中，我们的参数是$W$和$b$，使用以下的策略更新参数：\n",
    "\n",
    "$$\n",
    "W := W - \\alpha \\frac{\\partial \\mathrm{loss}}{\\partial W}\n",
    "$$\n",
    "\n",
    "$$\n",
    "b := b - \\alpha \\frac{\\partial \\mathrm{loss}}{\\partial b}\n",
    "$$\n",
    "\n",
    "其中，$\\alpha$ 是学习率，一般设置为0.1，0.01等。\n",
    "\n",
    "接下来我们会求解损失函数对参数的偏导数。\n",
    "\n",
    "损失函数MSE记为：\n",
    "\n",
    "$$\n",
    "\\mathrm{loss}(y, Z) = \\frac{1}{n} \\sum^n_{i = 1} (y_i - Z_i)^2\n",
    "$$\n",
    "\n",
    "其中，$Z \\in \\mathbb{R}^{n}$是我们的预测值，也就是神经网络输出层的输出值。这里我们有$n$个样本，实际上是将$n$个样本的预测值与他们的真值相减，取平方后加和。\n",
    "\n",
    "我们计算损失函数对参数$W$的偏导数，根据链式法则，可以将偏导数拆成两项，分别求解后相乘：\n",
    "\n",
    "**这里我们以矩阵的形式写出推导过程，感兴趣的同学可以尝试使用单个样本进行推到，然后推广到矩阵形式**\n",
    "\n",
    "$$\\begin{aligned}\n",
    "\\frac{\\partial \\mathrm{loss}}{\\partial W} &= \\frac{\\partial \\mathrm{loss}}{\\partial Z} \\frac{\\partial Z}{\\partial W}\\\\\n",
    "&= - \\frac{2}{n} X^\\mathrm{T} (y - Z)\\\\\n",
    "&= \\frac{2}{n} X^\\mathrm{T} (Z - y)\n",
    "\\end{aligned}$$\n",
    "\n",
    "同理，求解损失函数对参数$b$的偏导数:\n",
    "\n",
    "$$\\begin{aligned}\n",
    "\\frac{\\partial \\mathrm{loss}}{\\partial b} &= \\frac{\\partial \\mathrm{loss}}{\\partial Z} \\frac{\\partial Z}{\\partial b}\\\\\n",
    "&= - \\frac{2}{n} \\sum^n_{i=1}(y_i - Z_i)\\\\\n",
    "&= \\frac{2}{n} \\sum^n_{i=1}(Z_i - y_i)\n",
    "\\end{aligned}$$\n",
    "\n",
    "**因为参数$b$对每个样本的损失值都有贡献，所以我们需要将所有样本的偏导数都加和。**\n",
    "\n",
    "其中，$\\frac{\\partial \\mathrm{loss}}{\\partial W} \\in \\mathbb{R}^{m}$，$\\frac{\\partial \\mathrm{loss}}{\\partial b} \\in \\mathbb{R}$，求解得到的梯度的维度与参数一致。\n",
    "\n",
    "完成上式两个梯度的计算后，就可以使用梯度下降法对参数进行更新了。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "训练神经网络的基本思路：\n",
    "\n",
    "1. 首先对参数进行初始化，对参数进行随机初始化（也就是取随机值）\n",
    "2. 将样本输入神经网络，计算神经网络预测值 $Z$\n",
    "3. 计算损失值MSE\n",
    "4. 通过 $Z$ 和 $y$ ，以及 $X$ ，计算参数的梯度\n",
    "5. 使用梯度下降更新参数\n",
    "6. 循环1-5步，**在反复迭代的过程中可以看到损失值不断减小的现象，如果没有下降说明出了问题**\n",
    "\n",
    "接下来我们来实现这个最简单的神经网络。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. 导入数据"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用kaggle房价数据，选3列作为特征"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "# 读取数据\n",
    "data = pd.read_csv('data/kaggle_house_price_prediction/kaggle_hourse_price_train.csv')\n",
    "\n",
    "# 使用这3列作为特征\n",
    "features = ['LotArea', 'BsmtUnfSF', 'GarageArea']\n",
    "target = 'SalePrice'\n",
    "data = data[features + [target]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. 数据预处理"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "40%做测试集，60%做训练集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "trainX, testX, trainY, testY = train_test_split(data[features], data[target], test_size = 0.4, random_state = 32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "训练集876个样本，3个特征，测试集584个样本，3个特征"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((876, 3), (876,), (584, 3), (584,))"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainX.shape, trainY.shape, testX.shape, testY.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. 参数初始化"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里，我们要初始化参数$W$和$b$，其中$W \\in \\mathbb{R}^m$，$b \\in \\mathbb{R}$，初始化的策略是将$W$初始化成一个随机数矩阵，参数$b$为0。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def initialize(m):\n",
    "    '''\n",
    "    参数初始化，将W初始化成一个随机向量，b是一个长度为1的向量\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    m: int, 特征数\n",
    "    \n",
    "    Returns\n",
    "    ----------\n",
    "    W: np.ndarray, shape = (m, ), 参数W\n",
    "    \n",
    "    b: np.ndarray, shape = (1, ), 参数b\n",
    "    \n",
    "    '''\n",
    "    \n",
    "    # 指定随机种子，这样生成的随机数就是固定的了，这样就可以与下面的测试样例进行比对\n",
    "    np.random.seed(32)\n",
    "    \n",
    "    W = np.random.normal(size = (m, )) * 0.01\n",
    "    \n",
    "    b = np.zeros((1, ))\n",
    "    \n",
    "    return W, b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. 前向传播"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里，我们要完成输入矩阵$X$在神经网络中的计算，也就是完成 $Z = XW + b$ 的计算。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def forward(X, W, b):\n",
    "    '''\n",
    "    前向传播，计算Z = XW + b\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    X: np.ndarray, shape = (n, m)，输入的数据\n",
    "    \n",
    "    W: np.ndarray, shape = (m, )，权重\n",
    "    \n",
    "    b: np.ndarray, shape = (1, )，偏置\n",
    "    \n",
    "    Returns\n",
    "    ----------\n",
    "    Z: np.ndarray, shape = (n, )，线性组合后的值\n",
    "    \n",
    "    '''\n",
    "    \n",
    "    # 完成Z = XW + b的计算\n",
    "    # YOUR CODE HERE\n",
    "    Z = np.dot(X,W)+ b\n",
    "    \n",
    "    return Z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-28.37377228144393\n"
     ]
    }
   ],
   "source": [
    "# 测试样例\n",
    "Wt, bt = initialize(trainX.shape[1])\n",
    "tmp = forward(trainX, Wt, bt)\n",
    "print(tmp.mean()) # -28.37377"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. 损失函数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来编写损失函数，我们以均方误差(MSE)作为损失函数，需要大家实现MSE的计算：\n",
    "\n",
    "$$\n",
    "\\mathrm{loss}(y, Z) = \\frac{1}{n} \\sum^n_{i = 1} (y_i - Z_i)^2\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def mse(y_true, y_pred):\n",
    "    '''\n",
    "    MSE，均方误差\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    y_true: np.ndarray, shape = (n, )，真值\n",
    "    \n",
    "    y_pred: np.ndarray, shape = (n, )，预测值\n",
    "    \n",
    "    Returns\n",
    "    ----------\n",
    "    loss: float，损失值\n",
    "    \n",
    "    '''\n",
    "    \n",
    "    # 计算MSE\n",
    "    # YOUR CODE HERE\n",
    "    n = y_pred.shape[0]\n",
    "    loss =sum((y_pred - y_true) * (y_pred - y_true)) / n\n",
    "    \n",
    "    return loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "39381033680.46006\n"
     ]
    }
   ],
   "source": [
    "# 测试样例\n",
    "Wt, bt = initialize(trainX.shape[1])\n",
    "tmp = mse(trainY, forward(trainX, Wt, bt))\n",
    "print(tmp) # 39381033680.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. 反向传播"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里我们要完成梯度的计算，也就是计算出损失函数对参数的偏导数的导数值：\n",
    "\n",
    "$$\n",
    "\\frac{\\partial \\mathrm{loss}}{\\partial W} = \\frac{2}{n} X^\\mathrm{T} (Z - y)\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\frac{\\partial \\mathrm{loss}}{\\partial b} = \\frac{2}{n} \\sum^n_{i=1}(Z_i - y_i)\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_gradient(X, Z, y_true):\n",
    "    '''\n",
    "    计算梯度\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    X: np.ndarray, shape = (n, m)，输入的数据\n",
    "    \n",
    "    Z: np.ndarray, shape = (n, )，线性组合后的值\n",
    "    \n",
    "    y_true: np.ndarray, shape = (n, )，真值\n",
    "    \n",
    "    Returns\n",
    "    ----------\n",
    "    dW, np.ndarray, shape = (m, ), 参数W的梯度\n",
    "    \n",
    "    db, np.ndarray, shape = (1, ), 参数b的梯度\n",
    "    \n",
    "    '''\n",
    "    \n",
    "    n = len(y_true)\n",
    "    \n",
    "    # 计算W的梯度\n",
    "    # YOUR CODE HERE\n",
    "    dW = np.dot(X.T,(Z - y_true)) * 2 / n\n",
    "    \n",
    "    # 计算b的梯度\n",
    "    # YOUR CODE HERE\n",
    "    db = sum(Z - y_true) * 2 / n \n",
    "    return dW, db"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(3,)\n",
      "-1532030241.2528903\n",
      "-364308.55576374085\n"
     ]
    }
   ],
   "source": [
    "# 测试样例\n",
    "Wt, bt = initialize(trainX.shape[1])\n",
    "Zt = forward(trainX, Wt, bt)\n",
    "dWt, dbt = compute_gradient(trainX, Zt, trainY)\n",
    "print(dWt.shape) # (3,)\n",
    "print(dWt.mean()) # -1532030241.25\n",
    "print(dbt) # -364308.555764"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. 梯度下降"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这部分需要实现梯度下降的函数\n",
    "$$\n",
    "W := W - \\alpha \\frac{\\partial \\mathrm{loss}}{\\partial W}\n",
    "$$\n",
    "\n",
    "$$\n",
    "b := b - \\alpha \\frac{\\partial \\mathrm{loss}}{\\partial b}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "def update(dW, db, W, b, learning_rate):\n",
    "    '''\n",
    "    梯度下降，参数更新，不需要返回值，W和b实际上是以引用的形式传入到函数内部，\n",
    "    函数内改变W和b会直接影响到它们本身，所以不需要返回值\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    dW, np.ndarray, shape = (m, ), 参数W的梯度\n",
    "    \n",
    "    db, np.ndarray, shape = (1, ), 参数b的梯度\n",
    "    \n",
    "    W: np.ndarray, shape = (m, )，权重\n",
    "    \n",
    "    b: np.ndarray, shape = (1, )，偏置\n",
    "    \n",
    "    learning_rate, float，学习率\n",
    "    \n",
    "    '''\n",
    "    # 更新W\n",
    "    W -= learning_rate * dW\n",
    "    \n",
    "    # 更新b\n",
    "    b -= learning_rate * db"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.004052439376931716\n",
      "0.0\n",
      "(3,)\n",
      "15320302.416581342\n",
      "3643.085557637409\n"
     ]
    }
   ],
   "source": [
    "# 测试样例\n",
    "Wt, bt = initialize(trainX.shape[1])\n",
    "print(Wt.mean()) # 0.00405243937693\n",
    "print(bt.mean()) # 0.0\n",
    "\n",
    "Zt = forward(trainX, Wt, bt)\n",
    "dWt, dbt = compute_gradient(trainX, Zt, trainY)\n",
    "update(dWt, dbt, Wt, bt, 0.01)\n",
    "\n",
    "print(Wt.shape) # (3,)\n",
    "print(Wt.mean()) # 15320302.4166\n",
    "print(bt.mean()) # 3643.08555764"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "完成整个参数更新的过程，先计算梯度，再更新参数，将compute_gradient和update组装在一起。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "def backward(X, Z, y_true, W, b, learning_rate):\n",
    "    '''\n",
    "    使用compute_gradient和update函数，先计算梯度，再更新参数\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    X: np.ndarray, shape = (n, m)，输入的数据\n",
    "    \n",
    "    Z: np.ndarray, shape = (n, )，线性组合后的值\n",
    "    \n",
    "    y_true: np.ndarray, shape = (n, )，真值\n",
    "    \n",
    "    W: np.ndarray, shape = (m, )，权重\n",
    "    \n",
    "    b: np.ndarray, shape = (1, )，偏置\n",
    "    \n",
    "    learning_rate, float，学习率\n",
    "    \n",
    "    '''\n",
    "    # 计算参数的梯度\n",
    "    # YOUR CODE HERE\n",
    "    dw,db = compute_gradient(X, Z, y_true)\n",
    "    # 更新参数\n",
    "    # YOUR CODE HERE\n",
    "    update(dw, db, W, b, learning_rate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.004052439376931716\n",
      "0.0\n",
      "(3,)\n",
      "15320302.416581342\n",
      "3643.085557637409\n"
     ]
    }
   ],
   "source": [
    "# 测试样例\n",
    "Wt, bt = initialize(trainX.shape[1])\n",
    "print(Wt.mean()) # 0.00405243937693\n",
    "print(bt.mean()) # 0.0\n",
    "\n",
    "Zt = forward(trainX, Wt, bt)\n",
    "backward(trainX, Zt, trainY, Wt, bt, 0.01)\n",
    "\n",
    "print(Wt.shape) # (3,)\n",
    "print(Wt.mean()) # 15320302.4166\n",
    "print(bt.mean()) # 3643.08555764"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. 训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(trainX, trainY, testX, testY, W, b, epochs, learning_rate = 0.01, verbose = False):\n",
    "    '''\n",
    "    训练，我们要迭代epochs次，每次迭代的过程中，做一次前向传播和一次反向传播，更新参数\n",
    "    同时记录训练集和测试集上的损失值，后面画图用。然后循环往复，直到达到最大迭代次数epochs\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    trainX: np.ndarray, shape = (n, m), 训练集\n",
    "    \n",
    "    trainY: np.ndarray, shape = (n, ), 训练集标记\n",
    "    \n",
    "    testX: np.ndarray, shape = (n_test, m)，测试集\n",
    "    \n",
    "    testY: np.ndarray, shape = (n_test, )，测试集的标记\n",
    "    \n",
    "    W: np.ndarray, shape = (m, )，参数W\n",
    "    \n",
    "    b: np.ndarray, shape = (1, )，参数b\n",
    "    \n",
    "    epochs: int, 要迭代的轮数\n",
    "    \n",
    "    learning_rate: float, default 0.01，学习率\n",
    "    \n",
    "    verbose: boolean, default False，是否打印损失值\n",
    "    \n",
    "    Returns\n",
    "    ----------\n",
    "    training_loss_list: list(float)，每迭代一次之后，训练集上的损失值\n",
    "    \n",
    "    testing_loss_list: list(float)，每迭代一次之后，测试集上的损失值\n",
    "    \n",
    "    '''\n",
    "    training_loss_list = []\n",
    "    testing_loss_list = []\n",
    "    \n",
    "    for epoch in range(epochs):\n",
    "        \n",
    "        # 这里我们要将神经网络的输出值保存起来，因为后面反向传播的时候需要这个值\n",
    "        Z = forward(trainX, W, b)\n",
    "        \n",
    "        # 计算训练集的损失值\n",
    "        training_loss = mse(trainY, Z)\n",
    "        \n",
    "        # 计算测试集的损失值        \n",
    "        testing_loss = mse(testY, forward(testX, W, b))\n",
    "        \n",
    "        # 将损失值存起来\n",
    "        training_loss_list.append(training_loss)\n",
    "        testing_loss_list.append(testing_loss)\n",
    "        \n",
    "        # 打印损失值，debug用\n",
    "        if verbose:\n",
    "            print('epoch %s training loss: %s'%(epoch+1, training_loss))\n",
    "            print('epoch %s testing loss: %s'%(epoch+1, testing_loss))\n",
    "            print()\n",
    "        \n",
    "        # 反向传播，参数更新\n",
    "        backward(trainX, Z, trainY, W, b, learning_rate)\n",
    "        \n",
    "    return training_loss_list, testing_loss_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.004052439376931716\n",
      "0.0\n",
      "[39381033680.46006, 3.390230782482135e+23]\n",
      "[38555252685.09385, 4.151607023181587e+23]\n",
      "-57055790600891.01\n",
      "-8824267814.59105\n"
     ]
    }
   ],
   "source": [
    "# 测试样例\n",
    "Wt, bt = initialize(trainX.shape[1])\n",
    "print(Wt.mean())          # 0.00405243937693\n",
    "print(bt.mean())          # 0.0\n",
    "\n",
    "training_loss_list, testing_loss_list = train(trainX, trainY, testX, testY, Wt, bt, 2, learning_rate = 0.01, verbose = False)\n",
    "\n",
    "print(training_loss_list) # [39381033680.460075, 3.3902307664083424e+23]\n",
    "print(testing_loss_list)  # [38555252685.093872, 4.1516070070405267e+23]\n",
    "print(Wt.mean())          # -5.70557904608e+13\n",
    "print(bt.mean())          # -8824267814.59"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9. 检查\n",
    "\n",
    "编写一个绘制损失值变化曲线的函数\n",
    "\n",
    "一般我们通过绘制损失函数的变化曲线来判断模型的拟合状态。\n",
    "\n",
    "一般来说，随着迭代轮数的增加，训练集的loss在下降，而测试集的loss在上升，这说明我们正在不断地让模型在训练集上表现得越来越好，在测试集上表现得越来越糟糕，这就是过拟合的体现。  \n",
    "\n",
    "如果训练集loss和测试集loss共同下降，这就是我们想要的结果，说明模型正在很好的学习。  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_loss_curve(training_loss_list, testing_loss_list):\n",
    "    '''\n",
    "    绘制损失值变化曲线\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    training_loss_list: list(float)，每迭代一次之后，训练集上的损失值\n",
    "    \n",
    "    testing_loss_list: list(float)，每迭代一次之后，测试集上的损失值\n",
    "    \n",
    "    '''\n",
    "    plt.figure(figsize = (10, 6))\n",
    "    plt.plot(training_loss_list, label = 'training loss')\n",
    "    plt.plot(testing_loss_list, label = 'testing loss')\n",
    "    plt.xlabel('epoch')\n",
    "    plt.ylabel('loss')\n",
    "    plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上面这些函数就是完成整个神经网络需要的函数了\n",
    "\n",
    "|函数名|功能|\n",
    "|-|-|\n",
    "|initialize | 参数初始化|\n",
    "|forward | 给定数据，计算神经网络的输出值|\n",
    "|mse | 给定真值，计算神经网络的预测值与真值之间的差距|\n",
    "|backward | 计算参数的梯度，并实现参数的更新|\n",
    "|compute_gradient | 计算参数的梯度|\n",
    "|update | 参数的更新|\n",
    "|backward | 计算参数梯度，并且更新参数|\n",
    "|train | 训练神经网络|\n",
    "|plot_loss_curve | 绘制损失函数的变化曲线|\n",
    "\n",
    "我们使用参数初始化函数和训练函数，完成神经网络的训练。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 1 training loss: 39381033680.46006\n",
      "epoch 1 testing loss: 38555252685.09385\n",
      "\n",
      "epoch 2 training loss: 3.390230782482135e+23\n",
      "epoch 2 testing loss: 4.151607023181587e+23\n",
      "\n",
      "epoch 3 training loss: 5.0555031043473346e+36\n",
      "epoch 3 testing loss: 6.193449607112222e+36\n",
      "\n",
      "epoch 4 training loss: 7.538764120634879e+49\n",
      "epoch 4 testing loss: 9.235676243151314e+49\n",
      "\n",
      "epoch 5 training loss: 1.1241801912459211e+63\n",
      "epoch 5 testing loss: 1.3772236581164478e+63\n",
      "\n",
      "epoch 6 training loss: 1.676377032318245e+76\n",
      "epoch 6 testing loss: 2.053715344583049e+76\n",
      "\n",
      "epoch 7 training loss: 2.4998127314176892e+89\n",
      "epoch 7 testing loss: 3.062499465288529e+89\n",
      "\n",
      "epoch 8 training loss: 3.727719702480177e+102\n",
      "epoch 8 testing loss: 4.566797925345699e+102\n",
      "\n",
      "epoch 9 training loss: 5.5587740655990125e+115\n",
      "epoch 9 testing loss: 6.810007161577351e+115\n",
      "\n",
      "epoch 10 training loss: 8.289241568194478e+128\n",
      "epoch 10 testing loss: 1.0155079839059102e+129\n",
      "\n",
      "epoch 11 training loss: 1.2360913569254583e+142\n",
      "epoch 11 testing loss: 1.5143250820573048e+142\n",
      "\n",
      "epoch 12 training loss: 1.843258915904197e+155\n",
      "epoch 12 testing loss: 2.258160930776424e+155\n",
      "\n",
      "epoch 13 training loss: 2.748666926618769e+168\n",
      "epoch 13 testing loss: 3.367368638150888e+168\n",
      "\n",
      "epoch 14 training loss: 4.098810974573118e+181\n",
      "epoch 14 testing loss: 5.0214187087646845e+181\n",
      "\n",
      "epoch 15 training loss: 6.112145215771052e+194\n",
      "epoch 15 testing loss: 7.487937484200716e+194\n",
      "\n",
      "epoch 16 training loss: 9.114428396533633e+207\n",
      "epoch 16 testing loss: 1.1166009253407125e+208\n",
      "\n",
      "epoch 17 training loss: 1.3591431823508963e+221\n",
      "epoch 17 testing loss: 1.6650748341615212e+221\n",
      "\n",
      "epoch 18 training loss: 2.0267537466567527e+234\n",
      "epoch 18 testing loss: 2.482958898240254e+234\n",
      "\n",
      "epoch 19 training loss: 3.0222943417058355e+247\n",
      "epoch 19 testing loss: 3.702587273475301e+247\n",
      "\n",
      "epoch 20 training loss: 4.506844061827736e+260\n",
      "epoch 20 testing loss: 5.521296597949067e+260\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 特征数m\n",
    "m = trainX.shape[1]\n",
    "\n",
    "# 参数初始化\n",
    "W, b = initialize(m)\n",
    "\n",
    "# 训练20轮，学习率为0.01\n",
    "training_loss_list, testing_loss_list = train(trainX, trainY, testX, testY, W, b, 20, learning_rate = 0.01, verbose = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "绘制损失值的变化曲线"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlcAAAF+CAYAAACvcD/nAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de5hcdZ3v+/e3L0kn3Z17yMXMEJztOEASQojIZVRgO5HLHHTUAUU86lbQuexxnnNgC4/7wNY/ZnQzo+gewQFl0CMbmBFxu4eMZlAY4TxySRARJBJgIsZ0qA7Q3anqS9Ldv/NHVXc6ne7cuqurVvX79Tz9dPWqVbV+tVjdfPL7/db3FyklJEmSNDnqKt0ASZKkWmK4kiRJmkSGK0mSpElkuJIkSZpEhitJkqRJZLiSJEmaRFUXriLitojIRcTTR7Dv/xURv4iIpyLihxFx/IjnfjsiNkXEs6V9Vpa2nxARj0bEtoi4OyJmlO/TSJKk6abqwhVwO3D+Ee77U2B9SmkN8G3gv4947pvADSmlE4HTgVxp++eBL6aU3gC8Bnx0MhotSZIEVRiuUko/Bl4duS0ificivh8RWyLioYj4vdK+D6SUuku7PQKsKO1/EtCQUvrX0n75lFJ3RARwHsUgBvAN4F3l/1SSJGm6qLpwNY5bgP+cUjoNuAq4aYx9Pgr8S+nx7wIdEfGdiPhpRNwQEfXAQqAjpdRf2m8H8Loyt12SJE0jDZVuwOFERAtwFvBPxY4nAGaO2udyYD3wttKmBuAtwKnAS8DdwIeB741xCNf/kSRJk6bqwxXF3rWOlNLasZ6MiLcDnwbellLqK23eAfw0pfRiaZ/vAmcAtwHzIqKh1Hu1AthZ7g8gSZKmj6ofFkwpdQH/HhF/DBBFp5Qenwr8PXBxSik34mWPA/MjYnHp5/OAX6TiKtUPAO8tbf8Q8L+m4GNIkqRpIop5o3pExJ3AOcAi4GXgeuBHwM3AMqARuCul9NmIuB9YDbSVXv5SSuni0vv8AfC3QABbgCtTSnsj4vXAXcACincbXj6ix0uSJGlCqi5cSZIkZVnVDwtKkiRlieFKkiRpElXV3YKLFi1KK1eurHQzJEmSDmvLli27U0qLR2+vqnC1cuVKNm/eXOlmSJIkHVZE/Gqs7Q4LSpIkTSLDlSRJ0iQyXEmSJE2iqppzNZZ9+/axY8cOent7K92UaampqYkVK1bQ2NhY6aZIkpQJVR+uduzYQWtrKytXrmTEws2aAiklXnnlFXbs2MEJJ5xQ6eZIkpQJVT8s2Nvby8KFCw1WFRARLFy40F5DSZKOQtWHK8BgVUGee0mSjk4mwlUldXR0cNNNNx3Tay+88EI6OjoOuc91113H/ffff0zvP9rKlSvZvXv3pLyXJEk6NoarwzhUuBoYGDjkazdu3Mi8efMOuc9nP/tZ3v72tx9z+yRJUnUxXB3GNddcwwsvvMDatWu5+uqrefDBBzn33HO57LLLWL16NQDvete7OO200zj55JO55ZZbhl871JO0fft2TjzxRK644gpOPvlkNmzYQE9PDwAf/vCH+fa3vz28//XXX8+6detYvXo1W7duBaC9vZ0/+IM/YN26dXz84x/n+OOPP2wP1Re+8AVWrVrFqlWruPHGGwEoFApcdNFFnHLKKaxatYq77757+DOedNJJrFmzhquuumpyT6AkSdNM1d8tONJn/vcz/GJn16S+50nL53D9/3HyuM9/7nOf4+mnn+bJJ58E4MEHH+Sxxx7j6aefHr6D7rbbbmPBggX09PTwpje9ife85z0sXLjwgPfZtm0bd955J7feeiuXXHIJ99xzD5dffvlBx1u0aBFPPPEEN910E3/zN3/D1772NT7zmc9w3nnnce211/L973//gAA3li1btvAP//APPProo6SUePOb38zb3vY2XnzxRZYvX859990HQGdnJ6+++ir33nsvW7duJSIOO4wpSZIOzZ6rY3D66acfUJrgy1/+MqeccgpnnHEGv/71r9m2bdtBrznhhBNYu3YtAKeddhrbt28f873f/e53H7TPww8/zPve9z4Azj//fObPn3/I9j388MP80R/9Ec3NzbS0tPDud7+bhx56iNWrV3P//ffzqU99ioceeoi5c+cyZ84cmpqa+NjHPsZ3vvMdZs+efbSnQ5Kk6tH+S3jhAUipYk3IVM/VoXqYplJzc/Pw4wcffJD777+fn/zkJ8yePZtzzjlnzNIFM2fOHH5cX18/PCw43n719fX09/cDxXpTR2O8/X/3d3+XLVu2sHHjRq699lo2bNjAddddx2OPPcYPf/hD7rrrLv7u7/6OH/3oR0d1PEmSqsYT34THvw6fbqtYE+y5OozW1lb27Nkz7vOdnZ3Mnz+f2bNns3XrVh555JFJb8Pv//7v84//+I8AbNq0iddee+2Q+7/1rW/lu9/9Lt3d3RQKBe69917e8pa3sHPnTmbPns3ll1/OVVddxRNPPEE+n6ezs5MLL7yQG2+8cXj4U5KkTMrnoGUxVLCUUKZ6riph4cKFnH322axatYoLLriAiy666IDnzz//fL761a+yZs0a3vjGN3LGGWdMehuuv/563v/+93P33Xfztre9jWXLltHa2jru/uvWrePDH/4wp59+OgAf+9jHOPXUU/nBD37A1VdfTV1dHY2Njdx8883s2bOHd77znfT29pJS4otf/OKkt1+SpClTyEHzcRVtQhztkFM5rV+/Pm3evPmAbc8++ywnnnhihVpUHfr6+qivr6ehoYGf/OQn/Mmf/MmU9jD530CSlBk3nQXzj4f331n2Q0XElpTS+tHb7bnKgJdeeolLLrmEwcFBZsyYwa233lrpJkmSVJ0KOVhxUN6ZUoarDHjDG97AT3/600o3Q5Kk6jY4AN2vQEtlhwWd0C5JkmpD9yuQBis+58pwJUmSakM+V/zesriizTBcSZKk2lAohSt7riRJkiZBvr343TlX1a2jo4ObbrrpmF9/44030t3dPfzzhRdeOCnr923fvp1Vq1ZN+H0kSaoZwz1XDgtWtckOVxs3bmTevHmT0TRJkjRSPgf1M6BpbkWbYbg6jGuuuYYXXniBtWvXcvXVVwNwww038KY3vYk1a9Zw/fXXA1AoFLjooos45ZRTWLVqFXfffTdf/vKX2blzJ+eeey7nnnsuACtXrmT37t1s376dE088kSuuuIKTTz6ZDRs2DK83+Pjjj7NmzRrOPPNMrr766sP2UPX29vKRj3yE1atXc+qpp/LAAw8A8Mwzz3D66aezdu1a1qxZw7Zt28ZspyRJNaHQXpxvVcGlbyBrda7+5RrY9fPJfc+lq+GCz4379Oc+9zmefvrp4YromzZtYtu2bTz22GOklLj44ov58Y9/THt7O8uXL+e+++4DimsOzp07ly984Qs88MADLFq06KD33rZtG3feeSe33norl1xyCffccw+XX345H/nIR7jllls466yzuOaaaw77Eb7yla8A8POf/5ytW7eyYcMGnnvuOb761a/yyU9+kg984APs3buXgYEBNm7ceFA7JUmqCUPrClaYPVdHadOmTWzatIlTTz2VdevWsXXrVrZt28bq1au5//77+dSnPsVDDz3E3LmH75I84YQTWLt2LQCnnXYa27dvp6Ojgz179nDWWWcBcNlllx32fR5++GE++MEPAvB7v/d7HH/88Tz33HOceeaZ/NVf/RWf//zn+dWvfsWsWbOOqZ2SJGVCFawrCFnruTpED9NUSSlx7bXX8vGPf/yg57Zs2cLGjRu59tpr2bBhA9ddd90h32vmzJnDj+vr6+np6eFY1noc7zWXXXYZb37zm7nvvvt4xzvewde+9jXOO++8o26nJEmZkG+HZadUuhX2XB1Oa2sre/bsGf75He94B7fddhv5fB6A3/zmN+RyOXbu3Mns2bO5/PLLueqqq3jiiSfGfP3hzJ8/n9bWVh555BEA7rrrrsO+5q1vfSt33HEHAM899xwvvfQSb3zjG3nxxRd5/etfz1/8xV9w8cUX89RTT43bTkmSMm1wcP+cqwrLVs9VBSxcuJCzzz6bVatWccEFF3DDDTfw7LPPcuaZZwLQ0tLCt771LZ5//nmuvvpq6urqaGxs5Oabbwbgyiuv5IILLmDZsmXDE80P5+tf/zpXXHEFzc3NnHPOOYcduvvTP/1TPvGJT7B69WoaGhq4/fbbmTlzJnfffTff+ta3aGxsZOnSpVx33XU8/vjjY7ZTkqRM63kN0kDFa1wBxLEMQ5XL+vXr0+bNmw/Y9uyzz3LiiSdWqEWVkc/naWlpAYoT6tva2vjSl75UsfZMx/8GkqSMyT0LN50B7/k6rH7vlBwyIraklNaP3m7PVRW67777+Ou//mv6+/s5/vjjuf322yvdJEmSqtvwuoKV77kyXFWhSy+9lEsvvbTSzZAkKTuGw9WSyrYDJ7RLkqRaUCVL30BGwlU1zQubbjz3kqRMyOegrhFmza90S6o/XDU1NfHKK6/4P/kKSCnxyiuv0NTUVOmmSJJ0aIX2Yq9VhZe+gQzMuVqxYgU7duygvb290k2ZlpqamlixYkWlmyFJ0qFVydI3UOZwFRHbgT3AANA/1u2Kh9PY2MgJJ5ww2U2TJEm1pEqWvoGp6bk6N6W0ewqOI0mSpqt8OyxZVelWABmYcyVJknRIKe2fc1UFyh2uErApIrZExJVj7RARV0bE5ojY7LwqSZJ01Hpeg8F9VVFAFMofrs5OKa0DLgD+LCLeOnqHlNItKaX1KaX1ixdXR+KUJEkZUih1zlTJnKuyhquU0s7S9xxwL3B6OY8nSZKmoeHq7NXRSVO2cBURzRHROvQY2AA8Xa7jSZKkaWq4Ont19FyV827BJcC9USzm1QD8z5TS98t4PEmSNB3lS8OCVTLnqmzhKqX0InBKud5fkiQJKPZcRT3MWlDplgCWYpAkSVmXz0HzIqirjlhTHa2QJEk6VoX2qplvBYYrSZKUdVW0riAYriRJUtbZcyVJkjRJUrLnSpIkadL0dcFAnz1XkiRJk2K4OvuSyrZjBMOVJEnKripb+gYMV5IkKcuqbOkbMFxJkqQsq7Klb8BwJUmSsqyQg6iD2Qsr3ZJhhitJkpRd+VwxWNXVV7olwwxXkiQpu6qsgCgYriRJUpZVWQFRMFxJkqQsK+TsuZIkSZoUKRXvFqyiOwXBcCVJkrJqbx76e6DZYUFJkqSJG67Obs+VJEnSxBVKBUSdcyVJkjQJqnBdQTBcSZKkrBpjXcHbHv533n3T/8fgYKpQowxXkiQpq4bWFWxeNLzpl7v28OvXeqiriwo1ynAlSZKyqpCDWQugvnF4U1tXL8vmNlWwUYYrSZKUVfncQXcK7ursYekcw5UkSdLRK7QfVOOqrbOX5fNmVahBRYYrSZKUTaN6rvJ9/ezp7Wepw4KSJEnHIJ+DliXDP+7q7AFwzpUkSdJR21uAfYUDhgXbOnsBnHMlSZJ01MZY+mYoXC2b65wrSZKkozPG0jdtHcVwtWTuzEq0aJjhSpIkZc8YS9/s6uphUcsMZjbUV6hRRYYrSZKUPWMsfdPW2VvxOwXBcCVJkrJoeOmbET1Xnb0snVPZ+VZguJIkSVlUyEHTPGiYMbxpZ0cPy+fZcyVJknT0RhUQLfT101UFBUTBcCVJkrKo0H7AfKtdXUNlGAxXkiRJRy+fO/BOweECos65kiRJOnqjeq52dhSXvnHOlSRJ0tHa1wt9XWP2XC2p8NI3YLiSJElZM1aNq65eFjTPoKmxsgVEwXAlSZKyZqjG1Yi7BYs1rirfawWGK0mSlDXjVGevhvlWYLiSJElZM8a6gm2dPVVR4woMV5IkKWtG9Vz17B2go3sfy+ZWvgwDGK4kSVLW5Nth5hxoLPZUDRUQnTZzriKiPiJ+GhH/XO5jSZKkaaBw4NI3bZ3FGlfLptGcq08Cz07BcSRJ0nSQzx04mb1jaOmbaTAsGBErgIuAr5XzOJIkaRoZvfTNNBsWvBH4L8DgeDtExJURsTkiNre3t5e5OZIkKfMKo3quOnuYN7uRWTMqX0AUyhiuIuIPgVxKacuh9ksp3ZJSWp9SWr948eJD7SpJkqa7/j7o7TyogGi1DAlCeXuuzgYujojtwF3AeRHxrTIeT5Ik1bpCaZSreX+HzM6OXpZVSY0rKGO4Sildm1JakVJaCbwP+FFK6fJyHU+SJE0DwwVER/RcdfVWTQFRsM6VJEnKkuGeq2K46t03wKuFvSyrksnsAA1TcZCU0oPAg1NxLEmSVMNGLX3z8tCdgvZcSZIkHYNRS9/sLNW4Wj5vekxolyRJmlz5dpjRAjNmA7Crq1id3Z4rSZKkY1HIHXCnYFtndRUQBcOVJEnKknzuoBpXc5oaaJ45JdPIj4jhSpIkZUeh/aAaV9U03woMV5IkKUtG91x19VTVfCswXEmSpKwY2Ac9rx6wrmBx6RvDlSRJ0tEr7C5+L9W46usfYHd+L0vnOCwoSZJ09EbVuHq5sw+AZfPsuZIkSTp6+dLSN6U5V22dxRpXDgtKkiQdi+Geq+Kw4K7S0jeGK0mSpGMxvK7gEmBEAdG5zrmSJEk6evkcNM6GmS1A8U7B1qYGWqqogCgYriRJUlaMWvpmZ0dP1Q0JguFKkiRlxUEFRHurbkgQDFeSJCkrCu0HFBBt6+xlWRUt2DzEcCVJkrIhnxsuILq3f5Dd+b6qq3EFhitJkpQFA/3Q/cr+AqJdvaRUfWUYwHAlSZKyoPsVIA3PuRqqceWcK0mSpGMxqoDoUI0re64kSZKOxXAB0VLPVZUufQOGK0mSlAWF0rqCpTlXOzt6aZnZQGtTYwUbNTbDlSRJqn7DPVeldQU7e1lahb1WYLiSJElZUMhB/UyYOQeAtq7eqhwSBMOVJEnKgnx7cb5VBFCcc7W0CguIguFKkiRlwYh1BfcNDJLb08eyedVXhgEMV5IkKQuGeq6A3J6+qi0gCoYrSZKUBSN6robKMDihXZIk6VgMDkJh93DPVTUXEAXDlSRJqnY9r0IaGK5x1dYxFK6ccyVJknT0RtW4auvsZfaMeuY0NVSwUeMzXEmSpOo2tK5gyxIAdnX1sHRuE1Eqy1BtDFeSJKm6DfVcNe+fc1Wt863AcCVJkqrdGEvfVOt8KzBcSZKkalfIQf0MaJpH/8AgL1fx0jdguJIkSdUu316scRVBe76PwVS9Na7AcCVJkqrdiAKi1V7jCgxXkiSp2uVzwwVEd3VWd40rMFxJkqRqV2gfvlNwZ0dx6Rt7riRJko7F4GAxXI24U7CpsY65sxor3LDxGa4kSVL16u2Awf79Na66imUYqrWAKBiuJElSNRuucbV/zlU1DwmC4UqSJFWzoaVvhu4W7Oip6jIMYLiSJEnVbETP1cBg4uU9fdO35yoimiLisYj4WUQ8ExGfKdexJElSjSq0F783H8fufB8Dg4mlVVyGAcrbc9UHnJdSOgVYC5wfEWeU8XiSJKnW5HMQ9TBr/nAB0eVV3nPVUK43TiklIF/6sbH0lcp1PEmSVIOGqrPX1dFWqnE1redcRUR9RDwJ5IB/TSk9Ws7jSZKkGpPfX+OqLQPV2aHM4SqlNJBSWgusAE6PiFWj94mIKyNic0Rsbm9vL2dzJElS1hRywzWudnX1MqOhjvmzq7eAKEzR3YIppQ7gQeD8MZ67JaW0PqW0fvHixVPRHEmSlBX59uEaV22lGlfVXEAUynu34OKImFd6PAt4O7C1XMeTJEk1JqX9c64o1riq9jIMUN6eq2XAAxHxFPA4xTlX/1zG40mSpFrS2wkDe6FlCTDUc1Xd862gvHcLPgWcWq73lyRJNW5EAdHBwcTLXb1Vf6cgWKFdkiRVqxFL3+wu9NE/mGpnWDAiPhkRc6Lo6xHxRERsKHfjJEnSNDai56qtIxtlGODIe67+U0qpC9gALAY+AnyubK2SJEkasfTN/hpXNdJzBQzd83gh8A8ppZ+N2CZJkjT58jmIOpi9gF2d2ajODkcerrZExCaK4eoHEdEKDJavWZIkador5GD2Iqirp62rlxn1dSyYPaPSrTqsI71b8KMUF19+MaXUHRELKA4NSpIklceIAqK7Oot3CtbVVf/A2ZH2XJ0J/DKl1BERlwP/FegsX7MkSdK0d0AB0WyUYYAjD1c3A90RcQrwX4BfAd8sW6skSZJGLn3TlY3q7HDk4ao/pZSAdwJfSil9CWgtX7MkSdK0NmLpm8HBxMudfZnpuTrSOVd7IuJa4IPAWyKiHqjuJaklSVJ29e2B/l5oOY5Xu/eyd2CQ5RmocQVH3nN1KdBHsd7VLuB1wA1la5UkSZreRta4KhUQzUrP1RGFq1KgugOYGxF/CPSmlJxzJUmSymO4Ovti2ko1rmpqzlVEXAI8BvwxcAnwaES8t5wNkyRJ09jwuoLHsasrWz1XRzrn6tPAm1JKOYCIWAzcD3y7XA2TJEnT2Mh1BTs7aKwPFjXPrGybjtCRzrmqGwpWJa8cxWslSZKOTqEdCJi9iLaOHpbMyUYBUTjynqvvR8QPgDtLP18KbCxPkyRJ0rSXz8HsBVDfQFtnb2bmW8ERhquU0tUR8R7gbIoLNt+SUrq3rC2TJEnTV6EdmktL33T1smbFvAo36Mgdac8VKaV7gHvK2BZJkqSifA5ajiOlRFtnL+efXCM9VxGxB0hjPQWklNKcsrRKkiRNb4UcrHgTrxb2srd/MDN3CsJhwlVKySVuJEnS1MvnigVEO4tlGLI058o7/iRJUnXpy8O+bmhZzK7OoRpX2Vj6BgxXkiSp2owoINrWZc+VJEnSxORL6wq2HEdbRw8NdcGilmwUEAXDlSRJqjbDPVfFYcElc5qoz0gBUTBcSZKkanPA0je9mbpTEAxXkiSp2hRKw4LNi9nVZbiSJEmamHwOZs0n1TWws6OH5YYrSZKkCSgUa1x1dO+jr38wU2UYwHAlSZKqTb59eL4VZKsMAxiuJElStSnkSvOtegCccyVJkjQho3quljssKEmSdIz29cDePdC8mLaOXurrgsWt2SkgCoYrSZJUTUbVuDqudWamCoiC4UqSJFWT4RpXx7Grqydz863AcCVJkqrJcM/VYto6ezM33woMV5IkqZqU1hVMpTlX9lxJkiRNRL44LNhVN5+efQOZq3EFhitJklRNCjlomktbdwKyV+MKDFeSJKma5ItL3+yvzu6cK0mSpGOXzxXLMHRkc+kbMFxJkqRqMrT0TWcPdUHmCoiC4UqSJFWTEUvfLG6dSWN99qJK9losSZJq075e6OssFRDtzeR8KzBcSZKkajFUnb1lMTs7ejI53woMV5IkqVqMLCDamc0ColDGcBURvxURD0TEsxHxTER8slzHkiRJNaBUQLR7xkK692azgCiUt+eqH/i/U0onAmcAfxYRJ5XxeJIkKctKPVcvD8wBYKlzrg6UUmpLKT1RerwHeBZ4XbmOJ0mSMq60aPOOvS0ALLfnanwRsRI4FXh0Ko4nSZIyqNAOM1rZWSj+6JyrcUREC3AP8Jcppa4xnr8yIjZHxOb29vZyN0eSJFWrfA5aipPZI+C4VsPVQSKikWKwuiOl9J2x9kkp3ZJSWp9SWr948eJyNkeSJFWzQnuxxlVnL4taZjKjIZtFDcp5t2AAXweeTSl9oVzHkSRJNaLUc7Wzsyez862gvD1XZwMfBM6LiCdLXxeW8XiSJCnLCrnhnquszrcCaCjXG6eUHgaiXO8vSZJqyMA+6HkNWorh6uz/sKjSLTpm2RzMlCRJtaW09E3vzAXs6evPdM+V4UqSJFVeqcbVazEPILPV2cFwJUmSqkGp5+rlgbkALMtodXYwXEmSpGpQ6rn6TX8rYM+VJEnSxJTWFdzeW1z65rg5MyvZmgkxXEmSpMrL56CxmR0FWNQyk5kN9ZVu0TEzXEmSpMobKiDa0ZvpIUEwXEmSpGpQIwVEwXAlSZKqQb4dWo6jrbPHnitJkqQJK+TYN2sRXb39mS7DAIYrSZJUaQP90P0qe+rnA9kuwwCGK0mSVGndu4HEq6Xq7M65kiRJmohSAdHc4BzAnitJkqSJKRUQ3bmvWJ19yZxsh6uGSjdAkiRNc/niuoLb+5pZ2NxIU2N2C4iC4UqSJFVaqefqhe5ZLJ2b/WjisKAkSaqsfA4amvj3rrrMz7cCw5UkSaq0QnuxOvuevszfKQiGK0mSVGn5HIPNi+jo3pf5AqJguJIkSZVWaKdnxkIg+2UYwHAlSZIqLZ8j31Cszu6woCRJ0kQMDkD37uHq7A4LSpIkTUT3q5AGyQ3OBRwWlCRJmpih6uz9rcyfnf0ComC4kiRJlZR/GYCX+lpYWgNDgmC4kiRJlVRa+ub57tk1MSQIhitJklRJpWHBX+ZnGa4kSZImLJ8j1c/gpe4Gw5UkSdKEFdoZmLUICOdcSZIkTVg+R+/M2qnODoYrSZJUSYUc+YYFgOFKkiRp4vLtw9XZa2HpGzBcSZKkShkchEI77YNzmDurkdkzGirdoklhuJIkSZXR8xqkAXb2t9bMkCAYriRJUqWUalz9em+z4UqSJGnC8sVw9UJ3c82UYQDDlSRJqpRCcembF3pqZ+kbMFxJkqRKKfVc7U5za+ZOQTBcSZKkSinkGIwGOnHOlSRJ0sTl2+mbuYBEHcuccyVJkjRBI6qzOywoSZI0UfkcHTGP1qYGWmbWRgFRMFxJkqRKKbSTS3Nrar4VGK4kSVIlpAT5HG39rTU13woMV5IkqRJ6XoPBfby0t8WeqyMVEbdFRC4ini7XMSRJUkaVCohu722uqcnsUN6eq9uB88v4/pIkKauGCojinKsjllL6MfBqud5fkiRlWGF/dXbnXE2yiLgyIjZHxOb29vZKN0eSJE2FfPH/+bu9W3DypZRuSSmtTymtX7x4caWbI0mSpkIhx2DU8xotzrmSJEmasHyOQv08mmfOoLWpsdKtmVSGK0mSNPUK7XTUzau5IUEobymGO4GfAG+MiB0R8dFyHUuSJGVMPsfuNLfmhgQByraQT0rp/eV6b0mSlHGFdtr6X2/PlSRJ0oSlRMrn2LGvhaU1VoYBDFeSJGmq9XURA320p7kst+dKkiRpgkbUuKrFOVeGK0mSNLUKI5e+cVhQkiRpYvL7l76x50qSJGmiCtouO78AAAr5SURBVMVhwe7GBcxpKlvhgooxXEmSpKmVzzFIMGPOYiKi0q2ZdIYrSZI0tQo5umIOS+a1VLolZWG4kiRJUyvfzm7m1eR8KzBcSZKkKZbyOXYNtNZkdXYwXEmSpCk2sGcX7ak2yzCA4UqSJE2llKgrtLM7zbXnSpIkacL25qkb6K3ZGldguJIkSVNpRAFRe64kSZImqlRAtKt+PnNnNVa4MeVhuJIkSVOn1HMVLcfVZAFRMFxJkqSpVFq0uWHukgo3pHwMV5Ikaerki8OCs+fXbriqvdUSJUlS1RrM5+hILSyd11rpppSNPVeSJGnK7O3cVdNlGMBwJUmSplB/18s1XYYBDFeSJGkKRaGd3dhzJUmSNClm9O5md5rL8hpdVxAMV5Ikaars7aZxoJvXYh7zZtdmAVEwXEmSpKlSqnHVP2tRzRYQBcOVJEmaKqUaV7Qsrmw7ysxwJUmSpkap56pxztIKN6S8DFeSJGlKDO4phqtZ85dVuCXlZbiSJElTovu1NgDmLKrtcOXyN5IkaUr0vtZGf2rmuPlzK92UsjJcSZKkKdHf9TIdNV6dHRwWlCRJU6WQYzeGK0mSpEkxo2c3rzKXBc0zKt2UsjJcSZKkKTFr36t0z1hY0wVEwXAlSZKmwr5eZg0W2Ne0qNItKTvDlSRJKr9SAdGo8ersYLiSJElTYKiAaEONV2cHw5UkSZoCe17ZCcCs+YYrSZKkCduzuxiuWhYur3BLys9wJUmSyq6no7j0zcLjVlS4JeVnuJIkSWXX3/UyXWkWSxbOq3RTys5wJUmSyi/fzivMZWGNFxAFw5UkSZoCjT3tdNXPp66utguIguFKkiRNgVl7X6WncWGlmzElDFeSJKnsWgdeY9+s2q/ODmUOVxFxfkT8MiKej4hrynksSZJUnVJ/H3PIk5prvzo7lDFcRUQ98BXgAuAk4P0RcVK5jidJkqpTR3uxxlXDnCUVbsnUaCjje58OPJ9SehEgIu4C3gn8oozHPKTHv/jHzCm8VKnDS5I0LTWmXuYDTfOXVbopU6Kc4ep1wK9H/LwDePPonSLiSuBKgN/+7d8uY3NgsGE2fQ3NZT2GJEk6UB/N/LRpBSvXnlvppkyJcoarse61TAdtSOkW4BaA9evXH/T8ZHrzf/5GOd9ekiSprBPadwC/NeLnFcDOMh5PkiSp4soZrh4H3hARJ0TEDOB9wPfKeDxJkqSKK9uwYEqpPyL+HPgBUA/cllJ6plzHkyRJqgblnHNFSmkjsLGcx5AkSaomVmiXJEmaRIYrSZKkSWS4kiRJmkSGK0mSpElkuJIkSZpEhitJkqRJZLiSJEmaRIYrSZKkSWS4kiRJmkSRUqp0G4ZFRDvwqzIfZhGwu8zHyArPRZHnYT/PxX6ei/08F0Weh/08F0XHp5QWj95YVeFqKkTE5pTS+kq3oxp4Loo8D/t5LvbzXOznuSjyPOznuTg0hwUlSZImkeFKkiRpEk3HcHVLpRtQRTwXRZ6H/TwX+3ku9vNcFHke9vNcHMK0m3MlSZJUTtOx50qSJKlsajZcRcT5EfHLiHg+Iq4Z4/mIiC+Xnn8qItZVop3lFBG/FREPRMSzEfFMRHxyjH3OiYjOiHiy9HVdJdo6FSJie0T8vPQ5N4/xfM1fEwAR8cYR/72fjIiuiPjLUfvU7HUREbdFRC4inh6xbUFE/GtEbCt9nz/Oaw/5dyVrxjkXN0TE1tLvwL0RMW+c1x7y9ylLxjkP/y0ifjPid+DCcV47Ha6Ju0ech+0R8eQ4r62Za2LCUko19wXUAy8ArwdmAD8DThq1z4XAvwABnAE8Wul2l+E8LAPWlR63As+NcR7OAf650m2dovOxHVh0iOdr/poY4zPXA7so1mqZFtcF8FZgHfD0iG3/Hbim9Pga4PPjnKtD/l3J2tc452ID0FB6/PmxzkXpuUP+PmXpa5zz8N+Aqw7zumlxTYx6/m+B62r9mpjoV632XJ0OPJ9SejGltBe4C3jnqH3eCXwzFT0CzIuIZVPd0HJKKbWllJ4oPd4DPAu8rrKtqmo1f02M4T8CL6SUyl28t2qklH4MvDpq8zuBb5QefwN41xgvPZK/K5ky1rlIKW1KKfWXfnwEWDHlDZti41wTR2JaXBNDIiKAS4A7p7RRGVSr4ep1wK9H/LyDg0PFkexTMyJiJXAq8OgYT58ZET+LiH+JiJOntGFTKwGbImJLRFw5xvPT6pooeR/j/6GcLtcFwJKUUhsU/1ECHDfGPtPx+vhPFHtzx3K436da8Oel4dHbxhkqnm7XxFuAl1NK28Z5fjpcE0ekVsNVjLFt9G2RR7JPTYiIFuAe4C9TSl2jnn6C4pDQKcD/AL471e2bQmenlNYBFwB/FhFvHfX8tLkmACJiBnAx8E9jPD2drosjNd2uj08D/cAd4+xyuN+nrLsZ+B1gLdBGcThstGl1TQDv59C9VrV+TRyxWg1XO4DfGvHzCmDnMeyTeRHRSDFY3ZFS+s7o51NKXSmlfOnxRqAxIhZNcTOnREppZ+l7DriXYpf+SNPimhjhAuCJlNLLo5+YTtdFyctDQ8Cl77kx9pk210dEfAj4Q+ADqTSZZrQj+H3KtJTSyymlgZTSIHArY3++6XRNNADvBu4eb59avyaORq2Gq8eBN0TECaV/nb8P+N6ofb4H/J+lO8TOADqHhgVqRWl8/OvAsymlL4yzz9LSfkTE6RSviVemrpVTIyKaI6J16DHFSbtPj9qt5q+JUcb9V+h0uS5G+B7wodLjDwH/a4x9juTvSuZFxPnAp4CLU0rd4+xzJL9PmTZqvuUfMfbnmxbXRMnbga0ppR1jPTkdromjUukZ9eX6onjn13MU7+T4dGnbJ4BPlB4H8JXS8z8H1le6zWU4B79PsYv6KeDJ0teFo87DnwPPULzL5RHgrEq3u0zn4vWlz/iz0uedltfEiPMxm2JYmjti27S4LigGyjZgH8Weh48CC4EfAttK3xeU9l0ObBzx2oP+rmT5a5xz8TzFeURDfzO+OvpcjPf7lNWvcc7D/1v6O/AUxcC0bLpeE6Xttw/9fRixb81eExP9skK7JEnSJKrVYUFJkqSKMFxJkiRNIsOVJEnSJDJcSZIkTSLDlSRJ0iQyXEma9iLinIj450q3Q1JtMFxJkiRNIsOVpMyIiMsj4rGIeDIi/j4i6iMiHxF/GxFPRMQPI2Jxad+1EfFIaeHde4cW3o2I/xAR95cWpX4iIn6n9PYtEfHtiNgaEXcMVaiXpKNluJKUCRFxInApxcVh1wIDwAeAZoprJK4D/g24vvSSbwKfSimtoVhpe2j7HcBXUnFR6rMoVqMGOBX4S+AkitWmzy77h5JUkxoq3QBJOkL/ETgNeLzUqTSL4gLLg+xfTPZbwHciYi4wL6X0b6Xt3wD+qbT22etSSvcCpJR6AUrv91gqrZsWEU8CK4GHy/+xJNUaw5WkrAjgGymlaw/YGPH/jNrvUGt6HWqor2/E4wH8+yjpGDksKCkrfgi8NyKOA4iIBRFxPMW/Y+8t7XMZ8HBKqRN4LSLeUtr+QeDfUkpdwI6IeFfpPWZGxOwp/RSSap7/MpOUCSmlX0TEfwU2RUQdsA/4M6AAnBwRW4BOivOyAD4EfLUUnl4EPlLa/kHg7yPis6X3+OMp/BiSpoFI6VA96JJU3SIin1JqqXQ7JGmIw4KSJEmTyJ4rSZKkSWTPlSRJ0iQyXEmSJE0iw5UkSdIkMlxJkiRNIsOVJEnSJDJcSZIkTaL/H+hK/kNiwuyjAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_loss_curve(training_loss_list, testing_loss_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通过打印损失的信息我们可以看到损失值持续上升，这就说明哪里出了问题。但是如果所有的测试样例都通过了，就说明我们的实现是没有问题的。运行下面的测试样例，观察哪里出了问题。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 0, W: [-0.00348894  0.00983703  0.00580923]\n",
      "epoch 0, b: [0.]\n",
      "\n",
      "dWt: [-4.18172940e+09 -2.19880296e+08 -1.94481031e+08]\n",
      "db: -364308.55576374085\n",
      "\n",
      "epoch 1, W: [41817293.96016916  2198802.97412493  1944810.31544993]\n",
      "epoch 1, b: [3643.08555764]\n"
     ]
    }
   ],
   "source": [
    "# 测试样例\n",
    "Wt, bt = initialize(trainX.shape[1])\n",
    "print('epoch 0, W:', Wt)  # [-0.00348894  0.00983703  0.00580923]\n",
    "print('epoch 0, b:', bt)  # [ 0.]\n",
    "print()\n",
    "\n",
    "Zt = forward(trainX, Wt, bt)\n",
    "dWt, dbt = compute_gradient(trainX, Zt, trainY)\n",
    "print('dWt:', dWt) # [ -4.18172940e+09  -2.19880296e+08  -1.94481031e+08]\n",
    "print('db:', dbt) # -364308.555764\n",
    "print()\n",
    "\n",
    "update(dWt, dbt, Wt, bt, 0.01)\n",
    "print('epoch 1, W:', Wt)  # [ 41817293.96016914   2198802.97412493   1944810.31544994]\n",
    "print('epoch 1, b:', bt)  # [ 3643.08555764]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到，我们最开始的参数都是在 $10^{-3}$ 这个数量级上，而第一轮迭代时计算出的梯度的数量级在 $10^8$ 左右，这就导致使用梯度下降更新的时候，让参数变成了 $10^6$ 这个数量级左右（学习率为0.01）。产生这样的问题的主要原因是：我们的原始数据 $X$ 没有经过适当的处理，直接扔到了神经网络中进行训练，导致在计算梯度时，由于 $X$ 的数量级过大，导致梯度的数量级变大，在参数更新时使得参数的数量级不断上升，导致参数无法收敛。\n",
    "\n",
    "解决的方法也很简单，对参数进行归一化处理，将其标准化，使均值为0，缩放到 $[-1, 1]$附近。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10. 标准化处理"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "标准化处理和第一题一样"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "stand = StandardScaler()\n",
    "trainX_normalized = stand.fit_transform(trainX)\n",
    "testX_normalized = stand.transform(testX)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "重新训练模型，这次我们迭代40轮，学习率设置为0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = trainX.shape[1]\n",
    "W, b = initialize(m)\n",
    "training_loss_list, testing_loss_list = train(trainX_normalized, trainY, testX_normalized, testY, W, b, 40, learning_rate = 0.1, verbose = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "打印损失值变化曲线"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmEAAAF+CAYAAADKnc2YAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXhd1X3v//f3HMmWZxlbYNnyAC4Y4hFwGMOYBLDhQkgJmUgDNw2hSVvSBBLo7Q+a9N42bdKU0AQIBDKUFEhDoGlwEsIYaJlsYwZjY4OxsWyD5XmUbems3x86doyRZMnW0bak9+t5znPO2Xvtra82+5E/rL322pFSQpIkSZ0rl3UBkiRJPZEhTJIkKQOGMEmSpAwYwiRJkjJgCJMkScqAIUySJCkDXTKERcQdEbEyIl5uQ9tTI2J2RDRExEV7rPt0RCwsvj5duoolSZLeqUuGMOBHwDltbPsmcCnw77svjIiDgOuB44HjgOsjYnDHlShJktSyLhnCUkq/B9bsviwixkbEbyJiVkQ8ERFHFtsuTim9CBT22M3ZwO9SSmtSSmuB39H2YCdJkrRfyrIuoAPdClyRUloYEccDNwFnttJ+BLB0t++1xWWSJEkl1y1CWET0B04C/iMidi7uvbfNmlnmM5wkSVKn6BYhjKbLqutSSlPasU0tcPpu32uAxzqwJkmSpBZ1yTFhe0opbQDeiIiPAESTyXvZ7LfAWRExuDgg/6ziMkmSpJLrkiEsIu4CngLGRURtRHwG+CTwmYh4AZgLXFBs+96IqAU+Anw/IuYCpJTWAH8HPFd8fb24TJIkqeQiJYdBSZIkdbYu2RMmSZLU1RnCJEmSMtDl7o4cOnRoGjNmTNZlSJIk7dWsWbNWpZSqmlvX5ULYmDFjmDlzZtZlSJIk7VVELGlpnZcjJUmSMmAIkyRJyoAhTJIkKQNdbkyYJEl6tx07dlBbW0t9fX3WpfRIFRUV1NTUUF5e3uZtSh7CIiIPzASWpZTO22NdAN8BpgNbgEtTSrNLXZMkSd1NbW0tAwYMYMyYMTT986rOklJi9erV1NbWcuihh7Z5u864HHklMK+FddOAw4uvy4GbO6EeSZK6nfr6eoYMGWIAy0BEMGTIkHb3QpY0hEVEDXAu8IMWmlwA/CQ1eRqojIjqUtYkSVJ3ZQDLzr4c+1L3hN0AfAUotLB+BLB0t++1xWXvEBGXR8TMiJhZV1fX8VVKkqT9sm7dOm666aZ92nb69OmsW7eu1TbXXXcdDz300D7tf09jxoxh1apVHbKv/VGyEBYR5wErU0qzWmvWzLJ3PVE8pXRrSmlqSmlqVVWzk85KkqQMtRbCGhsbW912xowZVFZWttrm61//Oh/4wAf2ub4DUSl7wk4Gzo+IxcDdwJkRcecebWqBkbt9rwGWl7AmSZJUAtdccw2vv/46U6ZM4eqrr+axxx7jjDPO4BOf+AQTJ04E4EMf+hDHHnss48eP59Zbb9217c6eqcWLF3PUUUfx2c9+lvHjx3PWWWexdetWAC699FJ+/vOf72p//fXXc8wxxzBx4kTmz58PQF1dHR/84Ac55phj+NznPsfo0aP32uP17W9/mwkTJjBhwgRuuOEGADZv3sy5557L5MmTmTBhAvfcc8+u3/E973kPkyZN4qqrrtrvY1ayuyNTStcC1wJExOnAVSmlS/Zo9kvgzyPibuB4YH1KaUWpapIkqSf42n/N5ZXlGzp0n+8ZPpDr/9f4Ftd/4xvf4OWXX2bOnDkAPPbYYzz77LO8/PLLu+4YvOOOOzjooIPYunUr733ve/njP/5jhgwZ8o79LFy4kLvuuovbbruNiy++mHvvvZdLLtkzPsDQoUOZPXs2N910E9/61rf4wQ9+wNe+9jXOPPNMrr32Wn7zm9+8I+g1Z9asWfzwhz/kmWeeIaXE8ccfz2mnncaiRYsYPnw4DzzwAADr169nzZo13HfffcyfP5+I2Ovl07bo9MlaI+KKiLii+HUGsAh4DbgN+Hxn17OnjfU7ePTVlazatC3rUiRJ6tKOO+64d0zZcOONNzJ58mROOOEEli5dysKFC9+1zaGHHsqUKVMAOPbYY1m8eHGz+/7whz/8rjZPPvkkH/vYxwA455xzGDx4cKv1Pfnkk1x44YX069eP/v378+EPf5gnnniCiRMn8tBDD/HVr36VJ554gkGDBjFw4EAqKir40z/9U37xi1/Qt2/f9h6Od+mUyVpTSo8BjxU/37Lb8gR8oTNqaKslq7dw2Q+f4+ZPHsO0id6oKUnqelrrsepM/fr12/X5scce46GHHuKpp56ib9++nH766c1O6dC7d+9dn/P5/K7LkS21y+fzNDQ0AE3zdbVHS+2POOIIZs2axYwZM7j22ms566yzuO6663j22Wd5+OGHufvuu/nud7/LI4880q6ftycfW7SHmsF9AFi2rvn/6JIk6d0GDBjAxo0bW1y/fv16Bg8eTN++fZk/fz5PP/10h9fwvve9j5/97GcAPPjgg6xdu7bV9qeeeir3338/W7ZsYfPmzdx3332ccsopLF++nL59+3LJJZdw1VVXMXv2bDZt2sT69euZPn06N9xww67LrvvDxxbtYVCfcvr1yhvCJElqhyFDhnDyySczYcIEpk2bxrnnnvuO9eeccw633HILkyZNYty4cZxwwgkdXsP111/Pxz/+ce655x5OO+00qqurGTBgQIvtjznmGC699FKOO+44AP70T/+Uo48+mt/+9rdcffXV5HI5ysvLufnmm9m4cSMXXHAB9fX1pJT4l3/5l/2uN9rbdZe1qVOnppkzZ5b0Z5z1L48zZkg/bv2TqSX9OZIkdZR58+Zx1FFHZV1GprZt20Y+n6esrIynnnqKP/uzP+uQHqu2au6/QUTMSik1GyjsCWvGiMo+9oRJktTFvPnmm1x88cUUCgV69erFbbfdlnVJrTKENWPE4D48v3T/bz2VJEmd5/DDD+f555/Puow2c2D+njat5OxN99N/6zI2b2vIuhpJktRNGcL2tLmOU177FkfHa16SlCRJJWMI29OgpqcojYhVLFtrCJMkSaVhCNtTxUAKvSubQpg9YZIkqUQMYc2IwSMZmTOESZLUVuvWreOmm27a5+1vuOEGtmzZsuv79OnTO+T5jIsXL2bChAn7vZ9SMIQ1IypHMzq/2suRkiS1UUeHsBkzZlBZWdkRpR2wDGHNGTSSalaybO2WvbeVJElcc801vP7660yZMoWrr74agG9+85u8973vZdKkSVx//fUAbN68mXPPPZfJkyczYcIE7rnnHm688UaWL1/OGWecwRlnnAHAmDFjWLVqFYsXL+aoo47is5/9LOPHj+ess87a9TzJ5557jkmTJnHiiSdy9dVX77XHq76+nssuu4yJEydy9NFH8+ijjwIwd+5cjjvuOKZMmcKkSZNYuHBhs3V2NOcJa07lKCrSNjatXZl1JZIktd+vr4G3XurYfQ6bCNO+0eLqb3zjG7z88su7Zqh/8MEHWbhwIc8++ywpJc4//3x+//vfU1dXx/Dhw3nggQeApmdKDho0iG9/+9s8+uijDB069F37XrhwIXfddRe33XYbF198Mffeey+XXHIJl112GbfeeisnnXQS11xzzV5/he9973sAvPTSS8yfP5+zzjqLBQsWcMstt3DllVfyyU9+ku3bt9PY2MiMGTPeVWdHsyesOZVNd0j22lzL9oZCxsVIktT1PPjggzz44IMcffTRHHPMMcyfP5+FCxcyceJEHnroIb761a/yxBNPMGjQoL3u69BDD2XKlCkAHHvssSxevJh169axceNGTjrpJAA+8YlP7HU/Tz75JJ/61KcAOPLIIxk9ejQLFizgxBNP5O///u/5x3/8R5YsWUKfPn32qc72siesOTunqWAVb62vZ9SQvhkXJElSO7TSY9VZUkpce+21fO5zn3vXulmzZjFjxgyuvfZazjrrLK677rpW99W7d+9dn/P5PFu3bmVfnn3d0jaf+MQnOP7443nggQc4++yz+cEPfsCZZ57Z7jrby56w5lSOAmBE1FG7znFhkiTtzYABA9i4ceOu72effTZ33HEHmzZtAmDZsmWsXLmS5cuX07dvXy655BKuuuoqZs+e3ez2ezN48GAGDBjA008/DcDdd9+9121OPfVUfvrTnwKwYMEC3nzzTcaNG8eiRYs47LDD+Mu//EvOP/98XnzxxRbr7Ej2hDWnz2AK5f2oaVjF8nX1WVcjSdIBb8iQIZx88slMmDCBadOm8c1vfpN58+Zx4oknAtC/f3/uvPNOXnvtNa6++mpyuRzl5eXcfPPNAFx++eVMmzaN6urqXQPm9+b222/ns5/9LP369eP000/f6yXDz3/+81xxxRVMnDiRsrIyfvSjH9G7d2/uuece7rzzTsrLyxk2bBjXXXcdzz33XLN1dqTYl+68LE2dOjXNnDmz5D+n8L3jefitfrxy2ve58gOHl/znSZK0P+bNm8dRRx2VdRmdatOmTfTv3x9oujFgxYoVfOc738msnub+G0TErJTS1ObaezmyBbmdc4V5OVKSpAPSAw88wJQpU5gwYQJPPPEEf/M3f5N1Se3i5ciWVI5kePy3s+ZLknSA+uhHP8pHP/rRrMvYZ/aEtWTQSPqnTaxbsyrrSiRJUjdkCGtJ8Q7J3PpaCoWuNW5OktQzdbVx3t3Jvhx7Q1hLiiHskLSSVZu2ZVyMJEmtq6ioYPXq1QaxDKSUWL16NRUVFe3azjFhLdk1V9gqlq3bysED23dgJUnqTDU1NdTW1lJXV5d1KT1SRUUFNTU17drGENaSflUU8r0Z0dAUwo4eNTjriiRJalF5eTmHHnpo1mWoHbwc2ZIIGFRDTdSxbK13SEqSpI5lCGtFbvBoRuVXO02FJEnqcIaw1gwaSU2ssidMkiR1OENYaypHUpnWs2rt2qwrkSRJ3YwhrDWVowFI65ZmXIgkSepuDGGtGTQSgME73mb91h0ZFyNJkroTQ1hrinOF1UQdyx2cL0mSOlDJQlhEVETEsxHxQkTMjYivNdPm9IhYHxFziq/rSlXPPhkwjBRlTRO2OjhfkiR1oFJO1roNODOltCkiyoEnI+LXKaWn92j3RErpvBLWse9yeQoDh1Ozps5pKiRJUocqWU9YarKp+LW8+OpyD7TKDR5NTW6VIUySJHWoko4Ji4h8RMwBVgK/Syk900yzE4uXLH8dEeNLWc++iMpRjMyt9nKkJEnqUCUNYSmlxpTSFKAGOC4iJuzRZDYwOqU0GfhX4P7m9hMRl0fEzIiY2ekPJh00kqq0hrfWbujcnytJkrq1Trk7MqW0DngMOGeP5Rt2XrJMKc0AyiNiaDPb35pSmppSmlpVVdUZJf9B8Q7JxrXOFSZJkjpOKe+OrIqIyuLnPsAHgPl7tBkWEVH8fFyxntWlqmmfVDbNFdZ363LqdzRmXIwkSeouSnl3ZDXw44jI0xSufpZS+lVEXAGQUroFuAj4s4hoALYCH0spHViD93ebK2zF+noOHdov44IkSVJ3ULIQllJ6ETi6meW37Pb5u8B3S1VDhxg4ghS5XXOFGcIkSVJHcMb8vcmX09hvGDVRx7J1W7KuRpIkdROGsDbIDR5FjbPmS5KkDmQIa4Nc5ShG5VZT64StkiSpgxjC2qJyJAezmhVrNu29rSRJUhsYwtqichR5CmxfW5t1JZIkqZswhLXFoKa5wnptWkZj4cCaQUOSJHVNhrC2KM4VVp1WsnJjfcbFSJKk7sAQ1haDagB2zRUmSZK0vwxhbVHeh4Y+VU0hzDskJUlSBzCEtVEMHkVN1FFrT5gkSeoAhrA2yg8exaj8anvCJElShzCEtdWgkVSziuVrNmddiSRJ6gYMYW1VOYpyGqhfuzzrSiRJUjdgCGur4jQVsaGWlJwrTJIk7R9DWFsVJ2ytanibdVt2ZFyMJEnq6gxhbVXZFMKcpkKSJHUEQ1hb9R5AQ+9KRjhNhSRJ6gCGsPaoHEWNPWGSJKkDGMLaIT94FDU5H10kSZL2nyGsHaKyadb8ZWudK0ySJO0fQ1h7VI6igu1sWrsy60okSVIXZwhrj+I0Fax/M9s6JElSl2cIa4/iNBUD61ewZXtDxsVIkqSuzBDWHsVZ80fEKpZ7h6QkSdoPhrD2qKiksbw/I2KVc4VJkqT9YghrjwgKg0Y23SFpT5gkSdoPhrB2KhtcnLDVnjBJkrQfDGHtFJVNE7Y6JkySJO0PQ1h7VY5iAFtYu6Yu60okSVIXZghrr+I0FWmtc4VJkqR9Zwhrr0FN01RUbFnGjsZCxsVIkqSuyhDWXsW5woazirfW12dcjCRJ6qoMYe3VbyiN+QpGxCqnqZAkSfusZCEsIioi4tmIeCEi5kbE15ppExFxY0S8FhEvRsQxpaqnw0TQOLCmaa4wp6mQJEn7qJQ9YduAM1NKk4EpwDkRccIebaYBhxdflwM3l7CeDpMfPMqeMEmStF9KFsJSk03Fr+XFV9qj2QXAT4ptnwYqI6K6VDV1lPzgUYzMrXauMEmStM9KOiYsIvIRMQdYCfwupfTMHk1GAEt3+15bXLbnfi6PiJkRMbOu7gCYn6tyFIPZwKo1a7KuRJIkdVElDWEppcaU0hSgBjguIibs0SSa26yZ/dyaUpqaUppaVVVVilLbpzhNRaNzhUmSpH3UKXdHppTWAY8B5+yxqhYYudv3GmB5Z9S0X4oTtpZtrCWld2VGSZKkvSrl3ZFVEVFZ/NwH+AAwf49mvwT+pHiX5AnA+pTSilLV1GGKc4UdUljJqk3bMy5GkiR1RWUl3Hc18OOIyNMU9n6WUvpVRFwBkFK6BZgBTAdeA7YAl5Wwno7TfxiFXPmuOySrBvTOuiJJktTFlCyEpZReBI5uZvktu31OwBdKVUPJ5HI09KumZkfTXGFTRlZmXZEkSepinDF/H+UOGs2IWOU0FZIkaZ8YwvZRfvAoamK1E7ZKkqR9YgjbR1E5moNjLW+tXp91KZIkqQsyhO2r4jQVO5wrTJIk7QND2L4a1BTCchuW7qWhJEnSuxnC9lVxrrCDdrzNxvodGRcjSZK6GkPYvho4nESOEVHn4HxJktRuhrB9lS9nR79h1MQqlq01hEmSpPYxhO2PylHOFSZJkvaJIWw/lB80ippYRa0hTJIktZMhbD/E4NEMizWsWLMx61IkSVIXYwjbH4NGkqdA/ZrarCuRJEldjCFsfxQnbI11zhUmSZLaxxC2PypHA9B/63K2NTRmXIwkSepKDGH7Y+AIAEbEKlasq8+4GEmS1JUYwvZHeQXb+1Q5TYUkSWo3Q9h+SoNGURN1TlMhSZLaxRC2n8oOGsWInLPmS5Kk9jGE7af84NGMiNUsX7s561IkSVIXYgjbX5UjKaeBzauXZV2JJEnqQgxh+2vQqKZ35wqTJEntYAjbX5VNIaxicy2FQsq4GEmS1FUYwvZXcdb8YamOlRu3ZVyMJEnqKgxh+6tXP7b3qmRErGKZ01RIkqQ2MoR1gEJxrrDFq7xDUpIktY0hrAP0GjKamtwqXlmxIetSJElSF2EI6wC5ylHUxGpeWbY+61IkSVIXYQjrCJWj6M02VqxYSkreISlJkvbOENYRindIDtz2loPzJUlSmxjCOkLlaADGxNvMXe64MEmStHeGsI4w9AhSrpzxuSWGMEmS1CaGsI5Q1os4+CiO7b2UVwxhkiSpDUoWwiJiZEQ8GhHzImJuRFzZTJvTI2J9RMwpvq4rVT0lVz2ZI1nEK8vWZV2JJEnqAspKuO8G4MsppdkRMQCYFRG/Sym9ske7J1JK55Wwjs5RPZn+z/8bbF7G2s3bGdyvV9YVSZKkA1jJesJSSitSSrOLnzcC84ARpfp5maueDMD43GInbZUkSXvVKWPCImIMcDTwTDOrT4yIFyLi1xExvjPqKYlDxpMix4TcYuYud9JWSZLUupKHsIjoD9wLfDGltGcX0WxgdEppMvCvwP0t7OPyiJgZETPr6upKW/C+6tWPGHI4x/R608H5kiRpr0oawiKinKYA9tOU0i/2XJ9S2pBS2lT8PAMoj4ihzbS7NaU0NaU0taqqqpQl75/qyYyPxU5TIUmS9qqUd0cGcDswL6X07RbaDCu2IyKOK9azulQ1lVz1JA5qXMW6umVs3d6YdTWSJOkAVsq7I08GPgW8FBFzisv+GhgFkFK6BbgI+LOIaAC2Ah9LXfnhi8XB+UfFYl59eyNTRlZmXJAkSTpQlSyEpZSeBGIvbb4LfLdUNXS6YZMAGB9LmLt8vSFMkiS1yBnzO1KfSlLlaKaUOy5MkiS1zhDWwaJ6MpPzS7xDUpIktcoQ1tGqJzGscQW1b71FY6HrDm+TJEmlZQjraNVTABjb8AaL6jZlXIwkSTpQGcI62s7B+T6+SJIktcIQ1tEGHELqP4xJeQfnS5KklhnCSiCqJzOl3McXSZKklhnCSqF6EqMal/L6spV05blnJUlS6RjCSqF6MjkKHFK/iBXr67OuRpIkHYAMYaWw++B8L0lKkqRmGMJKoXIUqaKSCbk3HJwvSZKaZQgrhQiiehJHly9l7vL1WVcjSZIOQIawUqmezNi0hFeXr8m6EkmSdAAyhJVK9RTK0w76rH+d9Vt2ZF2NJEk6wBjCSqU4OH9C7g1nzpckSe9iCCuVIWNJ5X0ZH4sdFyZJkt7FEFYquTwxbCJHO3O+JElqRptCWERcGREDo8ntETE7Is4qdXFdXvVkxrGYecvXZV2JJEk6wLS1J+x/p5Q2AGcBVcBlwDdKVlV3MWwSfdJWtte9Tv2OxqyrkSRJB5C2hrAovk8HfphSemG3ZWpJ9WQA3sMiFry9MeNiJEnSgaStIWxWRDxIUwj7bUQMAAqlK6ubqDqSlOvF+NwSx4VJkqR3KGtju88AU4BFKaUtEXEQTZck1ZqyXnDIUUxevpgZhjBJkrSbtvaEnQi8mlJaFxGXAH8DOO9CG0T1ZMbnljB3mYPzJUnSH7Q1hN0MbImIycBXgCXAT0pWVXcybBID0wbWv72YxkLKuhpJknSAaGsIa0gpJeAC4Dsppe8AA0pXVjdSPQWAsQ2vsXj15oyLkSRJB4q2hrCNEXEt8CnggYjIA+WlK6sbOWQ8KXIOzpckSe/Q1hD2UWAbTfOFvQWMAL5Zsqq6k159SUOOYGJuMXMNYZIkqahNIawYvH4KDIqI84D6lJJjwtooN3wyk/JLfIakJEnapa2PLboYeBb4CHAx8ExEXFTKwrqVYZMYmlazYtlSmobWSZKknq6t84T9H+C9KaWVABFRBTwE/LxUhXUrxZnzh9cvYOXGbRwysCLjgiRJUtbaOiYstzOAFa1ux7YaNhGACbHYS5KSJAloe5D6TUT8NiIujYhLgQeAGaUrq5vpU0mhcgzjc294h6QkSQLaeDkypXR1RPwxcDJND+6+NaV0X0kr62Zy1ZOYvP5ZfmUIkyRJtOOSYkrp3pTSl1JKf9WWABYRIyPi0YiYFxFzI+LKZtpERNwYEa9FxIsRcUx7f4Euo3oyNektlixfkXUlkiTpANBqT1hEbASau50vgJRSGtjK5g3Al1NKsyNiADArIn6XUnpltzbTgMOLr+NpejzS8e35BbqM4sz5A9bOZ0P92QyscK5bSZJ6slZ7wlJKA1JKA5t5DdhLACOltCKlNLv4eSMwj6ZJXnd3AfCT1ORpoDIiqvfj9zlwVU8CYELuDeZ5SVKSpB6vU+5wjIgxwNHAM3usGgEs3e17Le8OakTE5RExMyJm1tXVlarM0up/MI39hvGe3GJeWWEIkySppyt5CIuI/sC9wBdTSnumj2hmk3dd/kwp3ZpSmppSmlpVVVWKMjtFbvhkpuSX+PgiSZJU2hAWEeU0BbCfppR+0UyTWmDkbt9rgOWlrClLUT2ZQ1nGa8tW7r2xJEnq1koWwiIigNuBeSmlb7fQ7JfAnxTvkjwBWJ9S6r63D1ZPIkeBslXz2N5QyLoaSZKUobY+tmhfnAx8CngpIuYUl/01MAogpXQLTRO+TgdeA7YAl5WwnuwVH190ZFrEgrc3MmHEoIwLkiRJWSlZCEspPUnzY752b5OAL5SqhgPOoJE09q7kPQ1Ng/MNYZIk9Vw+/7EzRZAbPplJ+SU+vkiSpB7OENbJonoy4+JN5i9bnXUpkiQpQ4awzlY9mXIa2PHWPAqF5h5GIEmSegJDWGcrDs4/tOF13lyzJeNiJElSVgxhne2gsTSW9WV8LHbSVkmSejBDWGfL5YhhE5mYW8wrK9ZnXY0kScqIISwDueFTGJ9bwivL1mVdiiRJyoghLAvVk+hDPRuXv5p1JZIkKSOGsCwUB+dXb1nAyo31GRcjSZKyYAjLQtWRFHK9GJ9bzNxlDs6XJKknMoRlIV8OB7+Hibkl/H5hXdbVSJKkDBjCMpIbPonJZYt5+JW3aXqEpiRJ6kkMYVmpnkz/wkYa1i7l9bpNWVcjSZI6mSEsK9VTAJiSe42H5q3MuBhJktTZDGFZqZ4MFZVc2H8ujxjCJEnqcQxhWcmXw+Fn8b7CTGYvWcXazduzrkiSJHUiQ1iWxk2jT8N6prCAxxd4l6QkST2JISxLf/QBUq6c8yte4KF5b2ddjSRJ6kSGsCxVDCQOPYVp5bN5fEEdOxoLWVckSZI6iSEsa+OmU7V9KVXb3uS5xWuyrkaSJHUSQ1jWjjgHgLPLnvcuSUmSehBDWNYqR8KwSVzYdw4PzzeESZLUUxjCDgTjpnP4tldYv2oFi5w9X5KkHsEQdiAYN40gcWb+eR72kqQkST2CIexAUD0ZBo7gwr4vOlWFJEk9hCHsQBAB46bx3sbneWnJ26zfsiPriiRJUokZwg4U46bRq1DP8bzMYwu8JClJUndnCDtQjDmF1GsA5/WewyPeJSlJUrdnCDtQlPUm/uj9fCA/m8fnv02Ds+dLktStGcIOJOOmM6hhNaO3vcqsJWuzrkaSJJWQIexAcvgHSZHn7LLZTtwqSVI3Zwg7kPQ9iBh1IudVvMDDTlUhSVK3VrIQFhF3RMTKiHi5hfWnR8T6iJhTfF1Xqlq6lCOnM2rHG2xb9QaLV23OuhpJklQipewJ+xFwzl7aPJFSmlJ8fb2EtXQd46YB8MHcLC9JSpLUjZUshKWUfg+sKdX+u57tzp4AABeLSURBVK2DDoOqIznfS5KSJHVrWY8JOzEiXoiIX0fE+IxrOXCMm8bkxrnMf+NNNtQ7e74kSd1RliFsNjA6pTQZ+Ffg/pYaRsTlETEzImbW1dV1WoGZGXcuORp5H3P4/YIe8PtKktQDZRbCUkobUkqbip9nAOURMbSFtremlKamlKZWVVV1ap2ZGHEsqV8V03s9zyPzHBcmSVJ3lFkIi4hhERHFz8cVa1mdVT0HlFyOOOIcTsu9wBPzl9FYSFlXJEmSOlgpp6i4C3gKGBcRtRHxmYi4IiKuKDa5CHg5Il4AbgQ+llIybew0bjp9Cps5YttLPP+ms+dLktTdlJVqxymlj+9l/XeB75bq53d5h51OKuvD2Y2zeGjeSqaOOSjriiRJUgfK+u5ItaRXX2LsGUzrNYdH5r2VdTWSJKmDGcIOZOOmUdW4knzdKyxdsyXraiRJUgcyhB3IjjiHRDTNnu/ErZIkdSuGsANZ/4OJmvdybu/nfYSRJEndjCHsQDduGuMKr7N40QI2bWvIuhpJktRBDGEHunHTATiV2Tzh7PmSJHUbhrADXdU40kGHMa18Ng85e74kSd2GIexAF0GMm84JMZdn5y9x9nxJkroJQ1hXMG4aZWkH4+tnMWfpuqyrkSRJHcAQ1hWMPIFCxWDOys/ikflOVSFJUndgCOsK8mXkjjibD5bN4dFXVmRdjSRJ6gCGsK5i3DQGpI30XzmL2rXOni9JUldnCOsq/uj9pFwvPpCfzb2zlmVdjSRJ2k+GsK6i9wDi0FO4oGIOP/rvRWzZ7sStkiR1ZYawruTI6RzSsIwh9Yu557mlWVcjSZL2gyGsKxl3LuTK+NLgJ7nt94vY0VjIuiJJkrSPDGFdycBqmPwxzt72W7avf5tfzlmedUWSJGkfGcK6mvd9iVxhB18Z9BA3P/46BWfQlySpSzKEdTVDxhLjP8yHG39N3cq3eGiek7dKktQVGcK6olO+TFnDFq7s/zA3PfY6KdkbJklSV2MI64oOeQ8ceR6fjF/z2tLlPPPGmqwrkiRJ7WQI66pO+TK9d2zgc30f5abHXs+6GkmS1E6GsK5qxDEw9v18Jv9rnl1Qy8vL1mddkSRJagdDWFd26lX03bGGT/d+nFsetzdMkqSuxBDWlY0+CUafzBd6z+Chl95k8arNWVckSZLayBDW1Z3yZQZuX8lFZU/y/d8vyroaSZLURoawrm7smTD8GP6qzwPcP2sJKzfUZ12RJElqA0NYVxcBp17FkO3LOYf/5vb/fiPriiRJUhsYwrqDI6bBwe/hK/1m8O9PL2b91h1ZVyRJkvbCENYd5HJwypep3r6Yk3Y8zZ1PL8m6IkmStBeGsO5i/IVw0Fiu7f8r7nhiEfU7GrOuSJIktcIQ1l3k8nDKlxiz/TUm1j/Hf8xcmnVFkiSpFYaw7mTSR0mDarim33/x/cdfp6GxkHVFkiSpBSULYRFxR0SsjIiXW1gfEXFjRLwWES9GxDGlqqXHyJcTJ3+RI3fMo2bDbH714oqsK5IkSS0oZU/Yj4BzWlk/DTi8+LocuLmEtfQcR3+K1P8QvtLnv7j5sddJKWVdkSRJakbJQlhK6ffAmlaaXAD8JDV5GqiMiOpS1dNjlFcQJ/0FxzS+QN+Vs3lk/sqsK5IkSc3IckzYCGD30eO1xWXvEhGXR8TMiJhZV1fXKcV1acdeRuozmC8Xe8MkSdKBJ8sQFs0sa/baWUrp1pTS1JTS1KqqqhKX1Q307k+c8AXeV5jJljef57nFrXVISpKkLGQZwmqBkbt9rwGWZ1RL93PcZ0m9B/DFCnvDJEk6EGUZwn4J/EnxLskTgPUpJW/n6yh9KonjLueD6WmWvPo8L9Wuz7oiSZK0m1JOUXEX8BQwLiJqI+IzEXFFRFxRbDIDWAS8BtwGfL5UtfRYJ3weyvvwVxUP8MV7nmfL9oasK5IkSUVlpdpxSunje1mfgC+U6ucL6DeUOPYyzn3mFr656gKu/8/BfPMjk7OuSpIk4Yz53d9Jf0GU9+WuIbdz/6zF3P/8sqwrkiRJGMK6v4HVcMF3Gb7pZf7loF/wf+57iTdWbc66KkmSejxDWE8w/kNw/BWct+V+zsk/y5//+2y2NTRmXZUkST2aIayn+ODfwYhj+UbZ99m0YgH/MGN+1hVJktSjGcJ6irJe8JEfUZ4v4+7Km7nrfxbw4Ny3sq5KkqQeyxDWk1SOgg/fSvXWhdww6G6u/vmLLFu3NeuqJEnqkQxhPc0RZ8P7/opp237DtMLjXHnX8zQ0FrKuSpKkHscQ1hOd8Tcw+mT+b9ntrH/zJf7loQVZVyRJUo9jCOuJ8mVw0R2UVfTnzoHf40ePzeXJhauyrkqSpB7FENZTDRgGf3w7B297k+/0/zFfvPt56jZuy7oqSZJ6DENYT3bYacQZf80HdjzO9O2/4Us/m0OhkLKuSpKkHsEQ1tOdchWMfT/Xl/+YNa89x82Pv551RZIk9QiGsJ4ul4MP30auXxU/6v89bvvd88xcvCbrqiRJ6vYMYYJ+Q4iP/IihjSu5sc9tXHnX86zbsj3rqiRJ6tYMYWoy6njiA1/j1MZnmL7lPr7y8xdJyfFhkiSViiFMf3DiF+DI87i27C5WzXuCv/3lXBodqC9JUkkYwvQHEXDB94hBI/hh/5uY8dQLfO7fZrJle0PWlUmS1O0YwvROfSqJi3/MoLSRRyv/juWvzuSj33+alRvqs65MkqRuxRCmdxt+NPzvX9O/PPhl368zqu4xLrzpf3j1rY1ZVyZJUrdhCFPzhh8Nn32EsoPH8d3ct/jY9l9w0c3/zRML67KuTJKkbsEQppYNrIZLZxDjP8RfFP6Nb/f6Pp/74f9wz3NvZl2ZJEldniFMrevVFy76IZx+LR/c8Qj3D/hH/uneJ/mn38z3EUeSJO0HQ5j2LgJOvwY+8iMOb1zEQwO+xsOPP8pf3v089Tsas65OkqQuyRCmtht/IXHZDCorgv/q+3W2vvwrLvnBM6zZ7Oz6kiS1lyFM7TPiGOKzj9Dr4CP4Qa9vM3X5nXz4e0/yxqrNWVcmSVKXYghT+w0cDpf9mnjPBVyT/yl/tfVGLv7eYzzng78lSWozQ5j2zc4B+6ddwwXpUW6P/8uf3/Y7bnx4oTPsS5LUBoYw7btcDs64Fv74dibmFvFAn+t46uH7OONbj3HPc2/63ElJklphCNP+m3gRcekMhvbrxV29/h/fT3/Hnb/4T6Z/5wkefXUlKRnGJEnaU3S1fyCnTp2aZs6cmXUZas6Oeph5B+mJbxFbVvNY/iT+bsuFDBs7iWunHcWEEYOyrlCSpE4VEbNSSlObXWcIU4er3wBP30T6n38lbd/C/ZzOt7ZdyAlTJvHls8cxorJP1hVKktQpDGHKxuZV8MQ/k577AY0F+EnjB/l+4QIuPHkynz9jLAMryrOuUJKkkmothJV0TFhEnBMRr0bEaxFxTTPrT4+I9RExp/i6rpT1qJP1Gwrn/APxF7Mom3wxl+V/zeO9vkivJ7/JtH+cwQ//+w22NxSyrlKSpEyUrCcsIvLAAuCDQC3wHPDxlNIru7U5HbgqpXReW/drT1gXtnI+PPJ3MP9XrM8N4oZt5/PEoPP52Il/xP+aPJxDBlZkXaEkSR2qtZ6wshL+3OOA11JKi4pF3A1cALzS6lbqvg4+Ej72U6idycCH/pbrF/8bK+t/w7/95jQ+8esTOOTQiVwwZTjnjK9mUF8vVUqSurdShrARwNLdvtcCxzfT7sSIeAFYTlOv2NwS1qQDQc1U4tP/BYse5eDff4svLbmXL/NzXlsxhl8sPo6L7j+RQ8dN4oIpI3j/UQdTUZ7PumJJkjpcKUNYNLNsz2ufs4HRKaVNETEduB84/F07irgcuBxg1KhRHV2nshABY8+EsWcSG5bDK//J2Ln38ZWlP+Mr/Iz5iw7lP189nu+WncRR4ydzwZQRnDx2CGV5p7aTJHUPpRwTdiLwtymls4vfrwVIKf1DK9ssBqamlFa11MYxYd3c+lp45T9JL99HLHsOgLkcxi93HM9TFe/j6ElTOH/KcKaMHEw+11zOlyTpwJHJFBURUUbTwPz3A8toGpj/id0vN0bEMODtlFKKiOOAn9PUM9ZiUYawHmTdmzD3fgpz7yO3fDYAL6ax/KrhOOaUTaaiZgJTRh/M0aMHc8zIwY4jkyQdcDKbJ6x4ifEGIA/ckVL6fxFxBUBK6ZaI+HPgz4AGYCvwpZTS/7S2T0NYD7V2Mcy9n8aXf0H+rRcA2E45LxXGMKfwR8wpjGXdQZMYPvpIjhkzmGNHD+awof3J2VsmScqQk7Wqe1m3FGqfg2WzaFz6HKyYQ75xGwBrGMjzjWN5oTCWBeVHEDVTOerQURwzajCHH9Kfgwf0JsJgJknqHIYwdW+NO2DlK1A7k7RsFjuWPEf52oVE8T6QRYVq5qSxvF4Yzlv5YTQMGk2vqsOoqqpmzND+jBnajzFD+1LV34AmSepYhjD1PPUbYPnzsGwmO96cSaF2Nr23vvWOJhtSX95MB7MkHcyb6RDezg1j+6DR5IeMZdCwUYweOohhgyoY2r83VQN6M7hvL28GkCS1S1aTtUrZqRgIh50Gh53GruH627fAuiWw5g1Y+wb9Vy9ibN0iDlvzBr03zSafGmAjsBF2vJFnWRrK2wzmjTSQmWkgaxhIfa8hNPQZAv2qyA+oonflIQwYVEXVwKawNrR/bwb1KWdARRl9e+XtWZMktcgQpp6jV184+KimF00PTu2zc12hETYsh7VvwJo3yK15g6qVr3PQxpXE5lWU1y+k9451RCHBZppeK5s23ZHyrGEAq9Mg3k4DWUhfNqU+bI6+bM/3o7GsH429+lPoNYDoPYBcxUByfQZS3ncg5X0rqeg3kD4VvelTnqd3eZ4+xVfFzvdeuV2fy50nTZK6DUOYBJDLQ+XIptehp5IH+u3ZprEBtq6BzXVNr011pM0raVy/kor1bzNs40qqN9eR215HvmET5Q2b6N24pene3wZgS8s/fnvKU08vttGL+tSLbZRTTy820ov6VN60nHK205sdud405HqTcuUUcuUUcmWkKCflyyG3+3svIl8Ou97LyeXLIZ8nl8tDrozIlRH5MnK5PJEvL76XkSsrI3J5crkyIp8nF3kinyNyeSLy5PO5pm1zQT6Xb2qbz5MLyEcQEURAbtc7TctoWrZzeQQEQS7X9N70vWk57/geTfsoLqPYZmdP4x+2eed+dq6M4rd3LOed2/9h2W6f2bXTZte/e9todvmeWuogjVa26shO1X3ZV2u1dUV2Ugua/h5lOczEECa1Vb4M+h/c9CoKoKL4alahANs3wbaNf3htb3pP9RvYvmUD2zevp3HbZhq3b4UdW+m9Yyu9dtQzYMdWaKgnGurJNdaTa1hHrlBPWeM2ygrbyBd2kGtsIE+hE375tmlMQYEcCUi73oMCseud4i0TO9vtfKfYZuc2FN/TrvfisrRz+92WFX/+7tvRwrLdR8E23353717ffLt3t2lttG1z+2tteWv2bVRv5/yjsy+/T2fpWqOhVSora87inMtbnEO+5AxhUinlck3j0yoGvmtVAL2Lr/1SKEBhBzRub7pTtLH4ubDb553LCw2QGpveCwUoNFAoNFBo3O3VsPPzDgqNDaSUSIVGUipQKDRCobDrO4UCKTW+83uhEVIikSAVmj6nP3yGAqnQFK12LUuFpjakYht2+/yH96b9FGNZemesipR2+4e1+Okd+9pt+Z7rdv03aX39zkXRTAR759fWI1h7FrduHzbqtJuxDtyY8+7/fuqp+lcPyfTnG8Kkri6Xg1xvKNu3OJcrviRJncu/vZIkSRkwhEmSJGXAECZJkpQBQ5gkSVIGDGGSJEkZMIRJkiRlwBAmSZKUAUOYJElSBgxhkiRJGTCESZIkZcAQJkmSlAFDmCRJUgYMYZIkSRmIlFLWNbRLRNQBSzrhRw0FVnXCzzmQeQw8BuAxAI8BeAzAYwAeA2j/MRidUqpqbkWXC2GdJSJmppSmZl1HljwGHgPwGIDHADwG4DEAjwF07DHwcqQkSVIGDGGSJEkZMIS17NasCzgAeAw8BuAxAI8BeAzAYwAeA+jAY+CYMEmSpAzYEyZJkpQBQ9geIuKciHg1Il6LiGuyricLEbE4Il6KiDkRMTPrejpDRNwRESsj4uXdlh0UEb+LiIXF98FZ1lhqLRyDv42IZcVzYU5ETM+yxlKLiJER8WhEzIuIuRFxZXF5jzkXWjkGPeZciIiKiHg2Il4oHoOvFZf3pPOgpWPQY86DnSIiHxHPR8Svit877DzwcuRuIiIPLAA+CNQCzwEfTym9kmlhnSwiFgNTU0o9Zi6YiDgV2AT8JKU0objsn4A1KaVvFAP54JTSV7Oss5RaOAZ/C2xKKX0ry9o6S0RUA9UppdkRMQCYBXwIuJQeci60cgwupoecCxERQL+U0qaIKAeeBK4EPkzPOQ9aOgbn0EPOg50i4kvAVGBgSum8jvy3wZ6wdzoOeC2ltCiltB24G7gg45rUCVJKvwfW7LH4AuDHxc8/pukfom6rhWPQo6SUVqSUZhc/bwTmASPoQedCK8egx0hNNhW/lhdfiZ51HrR0DHqUiKgBzgV+sNviDjsPDGHvNAJYutv3WnrYH5+iBDwYEbMi4vKsi8nQISmlFdD0DxNwcMb1ZOXPI+LF4uXKbnv5ZU8RMQY4GniGHnou7HEMoAedC8VLUHOAlcDvUko97jxo4RhADzoPgBuArwCF3ZZ12HlgCHunaGZZj0v+wMkppWOAacAXipep1DPdDIwFpgArgH/OtpzOERH9gXuBL6aUNmRdTxaaOQY96lxIKTWmlKYANcBxETEh65o6WwvHoMecBxFxHrAypTSrVD/DEPZOtcDI3b7XAMszqiUzKaXlxfeVwH00Xabtid4ujo/ZOU5mZcb1dLqU0tvFP8QF4DZ6wLlQHP9yL/DTlNIviot71LnQ3DHoiecCQEppHfAYTWOhetR5sNPux6CHnQcnA+cXx0nfDZwZEXfSgeeBIeydngMOj4hDI6IX8DHglxnX1Kkiol9xMC4R0Q84C3i59a26rV8Cny5+/jTwnxnWkomdf2iKLqSbnwvFwci3A/NSSt/ebVWPORdaOgY96VyIiKqIqCx+7gN8AJhPzzoPmj0GPek8SCldm1KqSSmNoSkPPJJSuoQOPA/K9rvKbiSl1BARfw78FsgDd6SU5mZcVmc7BLiv6e8wZcC/p5R+k21JpRcRdwGnA0Mjoha4HvgG8LOI+AzwJvCR7CosvRaOwekRMYWmy/KLgc9lVmDnOBn4FPBScSwMwF/Ts86Flo7Bx3vQuVAN/Lh4x3wO+FlK6VcR8RQ95zxo6Rj8Ww86D1rSYX8PnKJCkiQpA16OlCRJyoAhTJIkKQOGMEmSpAwYwiRJkjJgCJMkScqAIUyS2igiTo+IX2Vdh6TuwRAmSZKUAUOYpG4nIi6JiGcjYk5EfL/4IOJNEfHPETE7Ih6OiKpi2ykR8XTxgcT37XwgcUT8UUQ8FBEvFLcZW9x9/4j4eUTMj4ifFmeYl6R2M4RJ6lYi4ijgozQ9iH4K0Ah8EugHzC4+nP5xmp4IAPAT4KsppUnAS7st/ynwvZTSZOAkmh5WDHA08EXgPcBhNM0wL0nt5mOLJHU37weOBZ4rdlL1oekBuwXgnmKbO4FfRMQgoDKl9Hhx+Y+B/yg+P3VESuk+gJRSPUBxf8+mlGqL3+cAY4AnS/9rSepuDGGSupsAfpxSuvYdCyP+vz3atfbMttYuMW7b7XMj/h2VtI+8HCmpu3kYuCgiDgaIiIMiYjRNf+8uKrb5BPBkSmk9sDYiTiku/xTweEppA1AbER8q7qN3RPTt1N9CUrfn/8FJ6lZSSq9ExN8AD0ZEDtgBfAHYDIyPiFnAeprGjQF8GrilGLIWAZcVl38K+H5EfL24j4904q8hqQeIlFrrkZek7iEiNqWU+mddhyTt5OVISZKkDNgTJkmSlAF7wiRJkjJgCJMkScqAIUySJCkDhjBJkqQMGMIkSZIyYAiTJEnKwP8P9dFqpt49WkoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_loss_curve(training_loss_list, testing_loss_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "计算测试集上的MSE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "60305.85267910155"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prediction = forward(testX_normalized, W, b)\n",
    "mse(testY, prediction) ** 0.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
